Village Iron Golems Wander off/despawn after a few minutes

After a few minutes of being at a village, the village iron golem disappears. and he doesnt re spawn either, I'm not sure if it despawns or wanders off, this has happened to me multiple times. Its pretty annoying to have to rebuild an iron golem every time this happens. and even once you rebuild it, he still wont attack people that attack the villagers. so mabye there could be a way to link an iron golem to a village.