When trying to connect to one of my friends worlds it simply says "Unable To Connect To World"

When I try to join my friends world it stays on connecting to multiplayer game, and after a minute or so it says "Unable To Connect To World". This does not happen when i join servers and it happens randomly. Lets just say Player A is hosting a world Player A invites me, i join. Then Player B gets invited, and they join. Player A then proceeds to go AFK, and I leave, but when Player B invites me back it says unable to connect to world, but in the past i have been able to connect to player b's invitation, so why can't i now? (Minecraft gives me no information about how or why this is happening, it just happens)