Weird world rendering when coming through nether portal

When going through my nether portal to my base the world loads really slowly and my frames drop down to around 10 from 150, but if I tp or go through an end portal to get home it renders as expected.

It seems to happen more frequently if I've loaded chunks in the Overworld somewhere else then gone through the nether and out somewhere else.