Unemployed Villagers take over claimed job blocks

After I had placed a smithing table in a village, a villager claimed the block as expected. Later I noticed that same villager wasn't restocking so I went back to the smithing table and I found a different villager who had taken over the other villager's block. I tried destroying the block but the second villager didn't lose his job (even though he was at novice level) so I tried replacing it and then a third villager claimed it!