Don't let me play with my friend

What happens is that when I join my friend's world I get "Failed to connect to the world" and the same thing happens to him when I join my world, we've been having this problem for several months now and we can't play together, except on the servers that let us get together