Despawning villagers

While making an underground (well lit) villager trading hall I noticed some of my villagers just despawning even though I had already traded with all of them (some even to max level), this happens both in normal and hard mode. All villagers have a bed and iron golems and cats spawn as well, hostile mobs can't spawn down there. They also can't escape their little working enovironments since there's fences around them and ladders that they can't reach.