Spectator mode - speeding up/down works only first time

Speeding up/down with mouse wheel works, but only once.

1. Go to spectator mode
2. Modulate speed with mouse wheel
3. Go to survival mode, and then back to spectator mode

Flying now becomes incredibly slow, and changing speed with mouse wheel doesn't work anymore. Reloading the world (and going into spectator mode again - exiting in spectator mode and going back puts you into some fake spectator mode... basically survival without HUD and damage, but I'm pretty sure someone found out already) fixes it.