I can't join my friends by invite, and can only connect when I use LAN.

When ever I try to join my friend with an invite, it doesn't work. We make sure we are on the same version, and we both have internet. All it does is load for a few minutes before saying that there was an error.