Villagers despawn after sleeping in a bed

I created a village in survival mode, fully lit and walled in.  Populated with a mix of cured zombie villagers and bred villagers.  All have accessible beds and workstations.  But I have found that sometimes (not always) if I sleep in or near the village when I wake up some of the villagers are gone. Appears random as to if it’s cured or bred that despawn. 

I have observed that when the villagers get out of their beds they teleport for a second similar to endermen.  I have found villagers standing on top of the wall at least 2 times in the morning, and there is no way for them to climb up there so I assume it’s the teleporting causing it.  I thought they might be teleporting underground so I filled in all the nearby caves just to be thorough.