Villagers despawning

I use to have at least 2 clerics, one with a journeyman rank, and at least 9 other villagers. Then I was going to a fishing spot right next to the village. When I came back, there were no clerics and there were only 5 villagers left in the village. To be honest, I was playing on peaceful and there was no way a zombie would have killed some of the villagers.