My villagers are disappearing.

I checked on my villagers, and there was only one of the 5. I looked around, came back, and there where two more. I did some more looking, came back, and all of my villagers were there. Then, I came back down the next in-game day, and only one villager was there. It was the same villager that was there the first time they disappeared.