Village Creation - Villagers not staying at newly built village

Put the summary of the bug you're having here

What I expected to happen was...: Villagers staying in the structure I built for them in my fortress. My fortress contains beds, doors, crafting tables and a bell.

What actually happened was...: Every time I bring a second villager from another village to my fortress, they pathfind back to their original village, which is hundreds of blocks away.