Redstone torch is updated but left floating when place on the side of a redstone block and the block is removed

I placed a redstone torch on the side of a redstone block. The torch was turned off when placed. I mined away the redstone block and the torch was left floating in the air, but it's signal had been returned to "on". This happened every time I did it in all directions.