Crossplay multiplayer world connection issues

My friend plays Minecraft on Xbox One and we tried to play together. He created a new world and we both have enabled mutiplayer and crossplay enabled and every time I tried to join his world game just said "Unable to connect to world". No error code or any explanation so I'm wondering if you could help us out.