Unable to connect to one my friend's world but can join others?

I'm trying to join my friend's world but it always ends up with the text "Unable To Connect". Weirdly enough my friend is having no issues inviting other players to his world and same for me. Why is this issue only happening between us? I would like a fast response as I'm doing a build challenge collaboration with him. Thank you for your time.