Villagers and iron golems randomly try to return to their original village location

Good day,

I was trading with villagers in a village nearby my base. I was getting tired of travelling back and forth, so I decided to relocate the villagers to a custom village I built nearby my base.

All was fine, but then iron golems, and soon after, villagers started loading in places that are far out from the custom village I built. They seem to be moving or trying to pathfind to the old village location. The old village location is far away, it's out of chunk loading, even with the render setting at max. The game or the villager AI seem to have their village location setting on the old village.

They could sleep in the new custom village, and they are working on work blocks in the custom village, but randomly it's like they are compelled to come back to the old village like there's something luring them back.