TAB button can't be used to close inventory

I just tried setting Open/Close Inventory button to TAB, this does work for opening the inventory. However when i try closing the inventory, the "Recepie Book" UI element gets selected instead, thus preventing the closing of the inventory menu.