Unable to break blocks in survival after switch in gamemode.

When I turn from creative to survival I can no longer destroy any kind of solid block in game mode - survival. However this does not seem to apply to blocks like torches. (This applies to all versions since I purchased Minecraft in around 1.2.5)