Attribute modifiers on items after swapping hands don't work

I noticed it's possible to circumvent any attribute modifiers from an item by swapping hands twice. Even though the modifier is set to be active in "any slot", it loses its effect after the swapping. This is a nuisance, as I can't effectively use these functions. An example item is a sword with enhanced damage, but with movementSpeed set below default. After double-tapping the swap button, movementSpeed returns to default.