Villager Losing Workstations

On a realm that I play on, whenever I try and link villagers to their workstations, they get assigned to it and then shortly after a different villager gets assigned to that specific workstation, which then unassigns the previous villager. I am not sure if this issue is caused due to the fact it is a realm, but It happens when I am right next to the villagers so it does not have anything to do with unloading the chunks/mobs. This also was an issue in the previous 1.11 version of the game.