Connection lost, you have been idle for to long! only in realms as admin

When flying around in my real in gamemode 1 after a certain amount of time I get this error

Connection lost
You been idle to long

Well i am flying right, doing admin tasks, so not being idle at all so this must be a mistake.

Steps to Reproduce:
1. Be the admin in a realm world
2. do /gamemode 1
3. fly around, sometimes stop flying
4. after a minute or 3-4 I get the error, sometimes it takes longer but it will happen.

When I log back in, I find myself in gamemode 0 so falling to the ground and probably die and lose all my stuff.