Game crashes with friend joins (Xbox)

When I have a texture pack on, and my friend try's to join first all it takes about 5-10 mins just stop start their texture pack download, but after it hits about 300mb/1.4gb my game will crash on Xbox series x. Happens every time they try to load in while I'm wanting for them to join.

Clarifications:
We are both on Xbox
Same house
Different Xbox account
I host with an active texture pack on survival
Friend does not own texture pack
My game force closes to the Xbox home screen(crash)
Not a realm just a lan.