Binding inventory key to mouse buttons does not allow the inventory to close without pressing ESC

When the inventory button is binded to its default button, E, the inventory can open and close normally via this button. When I rebind this to Mouse 4 (the button on the mouse that's off to the side by my thumb and closest to me), I can only open the inventory with this button and not close. This isn't like the other issues where they're binding to TAB, which is default the button to open player menu, this is a button that is not reserved for any other function by default. 

Code Analysis:

Code analysis by Apollo can be found in this comment.