Attacking villagers causes them to no longer be interacted with

Whenever I attack a villager too many times, all of the villagers can no longer be interacted with(including trading). The only way to fix this is by restarting the world. This also happens in Minecraft 1.10.2. Is there a way to fix this?