Off-hand hotkey doesn't work in inventories

The bug

Off-hand hotkey doesn't work in inventories. Expected behavior would be moving the item to off-hand slot or back, just like numbers 1-9 move to represented slots on hotbar or (ctrl+)q drops items.

How to reproduce

1. Open an inventory (player default, chest, crafting table, furnace, ...)
2. Hover on an item
3. Press the off hand key (default F)
   → Notice that nothing happens