Villagers don't remember bed or workstation

I have my villagers in an iron farm and it worked great until I went to the nether, then returned. When I came back to the farm, non of the villagers would sleep nor work, even though their beds and workstations were within reach. I had to destroy their beds/workstations and replace with new ones to get them to interact with them again.