Villagers disappearing

On the small server I play on with my friends, we have a completely walled-in town for our villagers, but they disappear. It's one at a time, every four or five days, but I need it fixed.