Significant FPS drop when being high above the world and looking down

The Bug

When being high above the world and looking downwards, the framerate drops significantly.
I discovered this while testing 1.17's new world height customization option. This does affect 1.16.4 as well though.
In my case, FPS drop from ~70 to ~30.

My guess is that despite the world being out of render distance vertically, the game still tries to render all the chunks below at once.

How To Reproduce

1. Be in creative mode and double tab space to start flying
2. /tp ~ 512 ~
3. Look down

Possibly relevant video settings

• Biome Blend 5x5
• Graphics: Fabulous
• Render Distance: 16 chunks
• Smooth Lighting: Maximum
• Max Framerate: Unlimited
• Use VSync: Off
• Clouds: Off
• Particles: All
• Mipmap Levels: 4
• Entity Shadows: On
• Entity Distance: 100%

Specs

4x Intel(R) Core(TM) i5-4590 CPU @3.30GHz
AMD Radeon (TM) R9 380 Series
16 GB RAM (2GB allocated to Minecraft)