Charging headset freezes game for a minute+

When plugging in my headset, the game will freeze for over a minute. This occurs when I first plug the headset in, or when the headset restarts itself during charging (this happens every ~five minutes).

These are the only relevant log outputs I get on single player:

20:31:18.181 Can't keep up! Is the server overloaded? Running 40081ms or 801 ticks behind
20:31:58.398 Can't keep up! Is the server overloaded? Running 24147ms or 482 ticks behind

This is an issue for me as I regularly use this headset wireless, and often need to charge it. It's very frustrating to have to regularly wait over a minute to continue playing