Same router multiplayer with my brother but disconnects minutes later every time

My brother likes to play Minecraft with other friends and I often want to join the server with him and his friends and after a while, the server disconnects. My brother hosts the server most of the time, so when he disconnects, everyone else does. When he plays with his friends, there is almost never a disconnection, but when I join his server, there will always be disconnection minutes or an hour later. This seems like an issue of internet overload, but I wonder if there is a fix for this sort of issue. I'm using my Xbox account and so is he. Although I believe the underlying issue stems from the router because we are on the same router and there is a disconnection every time I want to join his server. Please fix this bug. Thank you.