Baby villagers randomly despawn

I have a villager farm with different occupation villages in different locked rooms. When I go far away (for example, mining overnight) I will come back and any baby villagers respawn and appear as adults in different villager cells.