Unable to Join Servers after leaving a solo world

In order to replicate:

• Join solo world
• Walk around a bit
• Leave solo world
• Attempt to join a server

This eventually fixes itself given time but whenever I try to join a server after doing the above steps I get the "unable to connect to world" error.