sticky piston not retracting by inverted redstone signal.

I wanted to make a redstone contraption in which the sticky piston in the image retracts and pulls the block when I press the button.

What I expected to happen: I press the button, the redstone behind it will power the redstone torch, which unpowers the redstone behind that, which leaves the piston without redstone signal, making it retract, which makes the block attached to it move while the signal is gone.

What happened: I press the button, the redstone behind it will power the redstone torch, which unpowers the redstone behind that, which leaves the piston without redstone signal, making it retract for really only like...1 frame. The block attached to it does not move. (I actually caught the frame on image 3)

I attached some screenshots:
+ 1.png: button in default mode
+ 2.png: button activated
+ 3.png: button during the second "click" sound, so de-activating.

If I were to remove the redstone between the torch and the piston, the piston would retract normally.

Weird fact: I enter this place in my world via a latch or what is it called. Basically a button giving an inverted signal to a piston, but for an extended ammount of time. It...works normally.