World occasionally fails to load correctly

Sometimes, when you're just playing normally, some chunks fail to load, causing the entire chunk to be made of void_air. If you step in these "ghost chunks", you will glitch out because of a server-client disagreement about what block you're standing on. Mobs will also glitch out and spin rapidly without being able to move. This can be fixed temporarily by leaving the world and coming back, or pressing F3 + A to reload all chunks. This seems to happen more and more often as you move away from the last spot where the world was reloaded.

The potential reason for why this happens is because the server has trouble processing the new height limit changes, while the client is completely fine. The last time this happened was when singleplayer was changed to an internal server, so it is probably caused by a client-server disagreement of which chunks are trying to load.
Post-21w16a edit: this doesn't seem to be likely anymore, since it still happens eventhough the height changes were reverted.