off hand not stacking

When I was playing survival I had a stack of torches in my off hand and when I broke it from the wall and picked it didn't stack with my off hand it created a new stack as if I didn't have any torches.