Villagers unlink randomly from workstations

Basically, librarian villagers that I have trapped right next to their lecterns all randomly unlink from their workstation possibly when I leave the chunk, and they are very weird when I try to relink them, like they will link to the correct lectern, sleep, and in the morning wake up and be unlinked. There is no game mechanic that I can think of that would make this happen. Also I play on a realm. Thanks!