Unable to move items in the inventory

After some time of playing on my Survival server, inventory gets bugged. I can't move anything or craft; it acts as if I was Shift-clicking. I always have to exit Minecraft and launch it again. Not tested on Creative or Survival Singleplayer.