Cannot join Xbox friends' worlds on Windows 10

When I try to join my friends' world (not realm) I get stuck on "Connecting to multiplayer game" for several seconds before getting an "Unable to connect to world." message. Some of my other friends can join but others can't. I don't know if this relates, but I have the same Xbox Live account connected to both my phone (PE) and my computer, and both have Minecraft installed. Other people also can't join my world.