FPS drops when looking up in a tall world (4064 blocks)

The bug

If you generate a tall world using custom world files, looking up(or down when far away from min_y) decreases fps by a very high amount without there being a lot to render

To reproduce

1. Load the custom world in the attachments
2. Look up
   →  FPS drops dramatically

In 21w06a the total world height got increased to 384 by default, this issue now also occurs(less extreme, the world height is still lower than my custom world) by default in new worlds