Cant place objects with off hand when I have nothing in the main hand

Whenever I want to place a block or torch with the off hand, and I got nothing in the main hand, the block/torch appears for a moment but it is not placed, the item is not wasted. However, if I have something in my main hand, like a tool (sword), I can.