Minecraft switch from tv to mobile

If you have the console docked to the TV while playing with a controller that is not the joycons if you pick up the console it automaticly switches to joycons even tho they are not the primary controller shown in the controllers screen. Then using the connected controller (not attached joycons) prompts for a second player to join when only one controller shows in the input menu.   So switching from tv to tabletop is imposible without closing the game.   The game should always respect the controller shown in the