Buttons on Item Frames

It seems that if you mess around with buttons and Item frames they get really buggy. Only works if you put button on the actual block after item frame has been placed. I hope this makes sense.