Pistons move even though the signal is apart/removed

After this kind of order...

-Place a piston(or sticky piston, because this occurs to both) that is not looking up.
-Place any block except redstone block on top of it.
-Place a redstone block on top of that block.
-Remove the block which was on top of the piston. At this case, the piston was turned on.
-Remove the redstone block.

...the piston was still turned on.