Multiplayer game, players completely freeze and have to restart game

While playing multiplayer on beta 1.16.51 players other than the host completely freeze in place and they are then forced to restart the game. Seems to happen more often when opening shulker boxes but not totally sure. There is also alot of automated farms on the world but it happens even away from the farms