High FPS, but jittery lag spikes when loading new chunks.

If I turn Vsync off I can get 500 FPS but when I whenever I look around and loads new chunks the memory percentage goes all over the place and my FPS drops under 30 for around 1to 5 seconds. I have tried allocating more and less RAM and tried to do multiple fixes from other forums with the same problems but none of them help.