Middle Mouse Button does not work as a hotbar slot in inventory

I have my first hotbar slot set to the Middle Mouse Button, and yes, i have changed my pick block button to something different, now pressing the middle mouse button to go to slot 1 works, but in my inventory if I hover over an item and press middle mouse button to send it to my first slot, nothing happens. This is the same in the creative inventory too.