Custom inventory buttons not working

I was going to try to beat minecraft only using my mouse just for fun, but then i changed the inventory button to the middle mouse button, it opened normally but it only closed if i pressed ESC. All the keyboard keys work, but if i put it on any of my mouse buttons(my mouse is a 5 button mouse) it only closes if i press ESC.

To recreate it just press esc, go to options, controls and change the inventory button to any mouse button.