New villagers steal workstations from established villagers

This is happening in 1.19 (and was present in 1.18 too) on both bedrock and Java worlds, but only in hosted as far as we can tell. On a local world we do not seem to encounter this issue. 

I am trying to build a nice natural village where villagers live in their houses and go out to work, but any time I go a short distance from my village to do something, I come back and find too many employed villagers with insufficent job blocks, and they've all been swapping beds. As long as I stay in the village things seem OK, but I've got things to do!