Game switches window when you click bottom right inventory spot

After/while interacting with villagers, if I click the bottom right inventory slot (the one right above the rightmost hotbar spot), the game occasionally switches out of the tab. It can be fixed by exiting the villager trade screen, and manually switching tabs then switching back to the game, but it's frustrating