Armor default attribute modifier component can't be removed

Removing the default attribute modifier component from armor with the "!"prefix in a give command or loot table, does not remove the attribute modifiers that are on armor by default.

To reproduce, run the command:

/give @s minecraft:iron_chestplate[!minecraft:attribute_modifiers]

This will give a player an iron chestplate that still says +6 armor, when the expected behavior is that it gives no armor.