Villagers disappearing after a raid

I After a successful defend against Pillagers, I headed back home to get trading items, came back to ghost village. All villagers and the Iron Golem disappeared. Then i started exploring the rest of the map losing the render distance of village. After i came back every villager was there, even the Iron Golem. Again i headed back home for items, when another raid began. I defended the village again, and of course everyone again despawned.