Unable to connect to world, unless I am not in my house

My brother and I always try to play Minecraft together, but we're always Unable to connect to each others worlds. We can play servers just fine, and the strangest thing is, this issue only comes up when we're at home. In other buildings, we can join each other without issues.

 

Also, even when I'm at home, I can join other friends that are at their homes, and my brother can join other people as well.

 

This has been happening for years. Like I said, it only happens in our house, and when we're in other buildings, it seems to connect fine. When we're at home, we can also connect to other people.