Villagers wander to their village if moved

I recently started a new survival world. When I tried to obtain villagers for trading, they instantly headed back to their village as soon as I destroyed the boat or minecart.
 Even when I put them inside of closed off rooms, they pushed against the wall instead of getting employed or sleeping.