Xbox Controller - Cursor Constantly Moves Left in all Menus

It's fine starting up the game but, occasionally after playing for long periods the cursor will get stuck moving left making it very difficult to navigate to do anything. and the only way to fix is a game restart. 

Im sure this isnt my controller because it still occurs with the analog stick and d-pad held in place, and it doesn't occur on the Xbox Home screen.