Villagers despawning

I had a building filled with villagers and randomly after playing for about 3 hours I walked away and went back and they were all gone. A lot of them were diamond level and had beds and workstations. I want to start a new villager trading center but am afraid to spend the time and lose it again