Armorer villagers not restocking due to job block placement

After some testing if you put the job block in a different location it will restock, It only doesn't restock when the Job block is in that spot. Edit: The time and conditions to cause a restock were in effect.