Spectators can continue breaking blocks

When a survival player is breaking a block, then switches to spectator with F3 + N, the spectator can continue to break the block even in spectator mode. Instead of destroying it, though, the block resets to breaking stage 0. When the player goes into survival again, the block gets instantly broken.