Breaking job blocks previously linked to villagers causes the game to crash

Breaking villager job site blocks causes the game to crash. I broke a grindstone which was linked to a villager and my game crashed. I tried it with lecterns and blast furnaces and the same thing happened