Item frame comparator not outputting if frame on same block with torch

Tried to make a hidden door in my castle that would be activated by rotating a wall (my torch 'holder') inside the item frame enough times. However, it sems that having a torch on the same side of the block as the item frame completely stops the comparator from working.

It should be noted that the comparator stays on if I add the torch after rotating the fence, but will go to zero the next time the wall is turned afterwards.

Cheers,
Lynx