Tabing to a different screen (Win10) causes connected xbox players to loose connection to server

My partner and I are both in the Better Together Beta, they are hosting a game via Windows 10 while I connect via Xbox one. If they Alt+Tab to another window, like the Xbox App, it disconnects me from the server.