All chunks reload when entering spectator mode

The bug

When you switch between spectator mode and any other gamemode, the entire world unloads and re-tesselates over the course of several seconds. This big white flash is generally very distracting. It makes it really difficult to use spectator mode for adventure map cutscenes effectively without flashbanging your players.

How to reproduce

1. Fly up really high
/gamemode spectator

Expected result
You enter spectator mode and that's it

Actual result
You enter spectator mode and also the entire world reloads