Can't interact with mouse on Xbox

Using a mouse on console SOMEWHAT works SOME of the time. I can still break and place blocks as well as look around. But whenever I try to interact with any inventory, I can't pick up or move stuff, and I can't interact with any menu items. The only way around this is clicking the enter key on my keyboard rather than left click.

Haven't been able to investigate on any other platforms.