Unable to pickup items held in off hand when the player's inventory is full

Say for example, I have a full inventory and am hold 32 torches in my off hand, if I walk over some torches that have been broken I cannot pick them up, BUT If I throw out something in my inventory and try to pick it up it will go straight to the torches in my off hand.