Entity positions of client and server are out of sync

I was building a carrot farm and my farmer was around (some blocks away and visible).

I removed a piece of farmland and tried several times to place a dirt block but they all instantly disappeared (server-client disagreement).

So I decided to exit the map and after reloading, my farmer appeared where I tried to place the dirt.

It seems to be some kind of familiar with the invisible Mobs at MC-139122 but different.