Breaking a lot of instant-break blocks causes lag spikes.

I've experience this bug in a modded version of 1.9.4, then I went to the vanilla version to see if the bug still persisted, and it did to a lesser degree.

My setup:
CPU: i5-4690K (OCed to 4.5 GHz)
GPU: GTX 950
RAM: 16 GB

This happened in no other version of the game that I've played in, with my setup.

Video with FPS Graph: https://www.youtube.com/watch?v=NRwDqPs7Kss

Update: Heavier spikes are noticed, when my friends are building or breaking.

Update 2: Affecting all 1.9 versions, vanilla.