Items get grabbed when you open a chest

Sometimes, when I open a chest, the item in the 5th slot of the inventory (not the hotbar) gets grabbed as if you were to left click it.
I checked everything I could think of while getting this to work and I dont think it has anything to do with my mouse or touchpad.