Villager workstation AI

I placed a workstation near some villagers and one of them became the one I needed. Then suddenly they decide they don't want to work at that station anymore and another villager has connected to that workstation. The original villager won't change, because I already traded with him. It happened multiple times. Also when I enter the area, I get huge frame drops. That does not happen when I have a lot of unemployed villagers.