Joining in a friend's world problem

I have Xbox live account me and my brother, different accounts, and I can join in his world but he can't join in my world. We both can connect to servers and play though, and we have the same (newest) version and we have bought Minecraft! I don't know if this is a bug in version 1.14.32.1 Windows 10 Edition but can someone let me know. It says unable to connect to world. Thanks!