Villagers won't stay inside village.

While trying to turn a desert city into a chain of islands, I noticed that the villagers were just putzing around chatting with eachother in groups of 2 or 3, day & night, a considerable distance away from the buildings. They never farm & they don't go indoors at night. If I hit them, they'll move for a moment, then go back to where they were. Haven't seen any baby villagers.