Villager choosing a profession block that isn't near them

I build different things and I use a lot of barrels cause they look nice but I also have villagers and the villagers even if they are many blocks away from the barrel that is the only profession they will chose it's kind of annoying if you could please fix this in some way that would be great thanks