Xbox to Windows 10 Cross-play issues. Worlds.

Xbox players are never able to connect to my Windows 10 Worlds.

I am able to connect to their Xbox worlds occasionally, however still receive errors and am unable to join on certain days.

Error is always: Unable to Connect to World.

We can however all connect to my Realm most of the time.

Also, if I switch to my Xbox One (bedrock) version on the same network, we do not have issues connecting to each other's worlds.

There seems to be an issue with cross-play from Xbox to Windows 10 and vice versa when connecting to normal multiplayer worlds or I have some configuration issue however I've been doing alot of research and checking everything from firewall to router settings with no luck.

Please advise