Chunks do not unload on local multiplayer servers decreasing fps

When non-host players in a local multiplayer game load chunks fast (specifically elytra flight with fireworks) outside the render distance radius of the host player, chunks do not or take up to a minute to unload, even when the chunks are outside the render distance. This results in large amounts of lag that decreases the fps substantially (10-3 fps).