Teleported and View controls moved back by a few seconds with 600ms+ ping

On a few survival servers I play on, if I mine two or three blocks quickly, it teleports my location and view controls back to a few seconds ago, and causes me to break other things I don't want to. It is extremely hard to put up with, especially when mining.

To replicate the situation I have, 650ms ping is needed and a 6 GB RAM server.

1.14.4 gave me this issue, tested out 1.14.3 and it didn't happen then. Only 1.14.4 acts like this.

Server upgrades do nothing and I can't upgrade my internet for physical limits IRL.