Unable to connect to world message.

Everytime I try to join a friend's server, whether it's on Switch or any other console, I get the message 'Unable to connect to world'. I've managed to play crossplay a few times but I've only been able to play with people on Switch twice. Sometimes friends cannot join my world's either. I don't have this problem on any other game on Switch. I don't see anyone else having this problem either.