Villagers can still leave villages under some conditions

To reproduce, start a raid in a village with a bunch of baby villagers and all the baby villagers will end up outside the village along with some adults. Villagers may also leave villages under other conditions as well, but I haven't determined these other conditions even though I have seen villagers go out of their village for no reason.