Unable to connect to world.

I play on pc (windows  10 edition) and my friends play on Xbox. I can join my friends one time (on their world) and it works fine. When I get disconnected or when the host leaves, I cannot join any one of my Xbox friends until the person I want to join and I both restart our Minecraft games. This is really annoying and tedious. Is there a solution?