Villagers keep disappearing

On my friends realm, we had a naturally generated village. I spawn-proofed it and built a wall around it, so mobs couldn't get in. After a while, the librarian with mending disappeared. As far as I could tell, he was the only one that wasn't there anymore. He was named and there was no way they could have gotten out. 

Then, I transferred all the villagers into a new self-built village. There, I had not only built walls and lit up the whole area, but I had also built a roof and secured the entrance. Only players were able to access the area. The village grew even, a lot more baby-villagers spawned and two golems were spawned which the natural village had failed to do. But one day, I realized a few villagers were missing. I checked the surrounding places to see if they had somehow glitched out, but I was unable to find them. I didn't think much of it, because of the librarian that had previously disappeared. But today, I logged on again and I was left with one villager. I have no clue as to how the probably two dozen villagers disappeared.