Items take durability damage when attacking players with pvp off

Items take durability damage when you attack a player normally, but if pvp is turned off, you can't damage the player anymore, but the item still takes durability damage when you hit a player.

This happens when pvp is turned off in the server settings, as well as when pvp is turned off for teams (/scoreboard teams option <team> friendlyfire false).