Villagers randomly changes/assign/unassign work station

Villagers will unassign their job site and take another villagers work block. Trading seems to affect this. Might be related to a max lvl villager haveing it's work site block and changed, but cant confirm this.