Villagers teleporting to nearby bed

Villagers we put in a trading station keep teleporting randomly to nearby villagers with a bed. It is frustrating since they are difficult to manage. We got to the point that we broke the walls intentionally because of the random teleporting.