Most chunks unload when opening inventory

When you open your inventory, you can look through the transparency on the sides and see that a lot of chunks unload/disappear.

My render distance is bigger than what is shown while the inventory is open.