Can't join my friends minecraft world from different houses

I cant join my friends minecraft world from my house, but when I bring my xbox to his house I can, it is no wifi or internet connection problems and my NAT is fine as well multi-player, we can play on minecraft servers like mine plex but I can't join his world, and he can't join mine.