Game constantly fights over M&KB and Controller

Whenever I have both my XBox One controller AND my mouse and keyboard connected, the game constantly argues over which takes precedence rather than waits for me to start using either input and switch to that scheme. Basically, it looks like the XBox controls and the M&KB controls are constantly flickering because the game can't decide which scheme to use, even when I move my mouse or use my keyboard a bit. This also results in me having a difficult time selecting menus due to this.