Villager trade cursor jumps to different location when making selection

When trying to select trades with villagers the cursor likes to move over somewhere Els making the selection process miserable using a controller. This has been a problem for me for years now