Unable to swap inventory items fast enough

When swapping items in inventory slots, there is a delay.
Steps to reproduce:
1. Get two items
2. Grab the first item and hold it on top of the second item
3. Spam-click on the item
4. There will be a delay between swapping