Villagers take workstations from 400+ blocks away

I moved my villagers as far away from any workstations as possible to get some book trades and they STILL take on random workstations from my village over 400 blocks away. Makes good trading basically impossible.