Server doesn't always recognize the client is trying to break blocks, even in singleplayer

I have been having this issue since 20w51a. When I try to break a lot of sand or ice, sometimes they will glitch out and I can't break any of them. It is like every block I try to break becomes a ghost block.

I don't know how to reproduce it, since I have only seen it happen a few times.

In the video, I was using a Netherite Shovel with Efficiency V, Unbreaking III, Mending, and Silk Touch. But it also happens even when I am not instamining blocks.

(P.S. I know you don't need above Efficiency III on a Netherite Shovel)