it crashes every time i join my brothers game!!

Every time i join my brothers game minecraft just crashes and shuts itself down and i can't play it for a couple minutes and i have to restart my xbox and then it just keeps happening and it won't stop and its really annoying. Can you fix that or do something about that.