Game lags intensively when a controller is disconnected in local co-op

When playing Minecraft with my girlfriend, I usually disconnect my controller so I can play something on my laptop while she plays Minecraft on my world, however we've started to notice that after a minute or two since the "Controller Disconnected" prompt appears on my (player 1) screen, the game begins to lag from a stable 60fps to around 15fps. The lag however disappears completely after around 5 seconds once the controller is reconnected.

This has started to get very annoying, as most times I want to play other games while she plays Minecraft, but the experience becomes unbearable for her if I do.