Villagers in trading hall will randomly quit then take back their jobs on a realm

I have a trading hall with 30 windows for villager librarians (6 long 5 high and only the bottom 6 have villagers in them) all with a block above and below. Behind them, I have a glowstone on top and a lectern facing away from them (they are standing only in a 2 block hole). I have noticed that after I get all the enchantments I want, they will quit their jobs and then take them back switching all the trades again. It doesn't seem to happen consistently it just happens randomly. I tried to recreate this in single-player but I was unable to, so I think it only happens on a realm.