Switching between offhand and mainhand with an item that has attribute modifers stops working when swapped to mainhand

I have an item that when held reduces the players movement speed by 20%

/give @s carrot_on_a_stick{AttributeModifiers:[

{AttributeName:"generic.movement_speed",Name:"generic.movement_speed",Amount:-0.20,Operation:1,UUID:[I;578450971,-1780594405,-1263397648,-1384389364]}

]}

And when I hold it in the mainhand then swap it to offhand then go back to mainhand it doesn't reduce the movement speed as it should.