Nautilus Shells

Sometimes when fishing i'll press "y" on a Nautilus Shell in my hotbar and it will go to my offhand and will not let me remove it from my offhand unless I leave the game and come back.