Villagers disappearing

I have 4 villages in a map linked by rail. I switch between them for building and sometimes further away for exploiting. They are within a 1024x1024 square.

Two of those villages were naturally generated and two I built entirely from the scratch. In those two "artificial villages" villagers keep disappearing. I left one of them for a while, and realized there were only 6 villagers left. The village has 2 golems (I am pretty sure it had 3 at some point and one of them disappeared) so that's 14 villagers who vanished out of blue. This village has over 60 doors. In the other I built myself, I had 4 villagers, one of each occupation bar "clerk" and nitwit. Sometimes the librarian disappears and reappears, sometimes the farmer, sometimes the armor smith, but I never see more than 3 of them together at the same time.

I am playing in peaceful mode, the villages are encircled and all places where a villager could die from jumping are blocked for them