Villager does not change profession even though the job site block is destroyed

My villager became a cleric even though his job site block is outside of 32 blocks of range. Breaking the job site block had no effect on the villager's profession, so I had to kill the villager. This can also happen when you place the job site block near an unemployed villager while it is their wandering time at the beginning of the day