Villagers swapping and disconnecting from workstations

When setting up a villager trading hall on ps4 realms so lots of villagers locked side by side the villagers are swapping or disconnecting from their workstations whenever I leave the server  I have no idea of the affected versions is correct from my experience I've only seen this bug happen with farmers and librarians but I haven't tested it with the other villagers