Delay when switching items in inventory

There is a very noticeble delay when switching items in the inventory and hotbar. I have to wait a fraction of a second before I can replace the item I drag with the item that's in the item slot. This delay only exists for a certain period of time. After that, I can switch the items with no delay issues.

Please fix this delay, as it's extremely annoying when I want to switch items on servers in PvP games and it doesn't work because I close the inventory too fast because I assume that the item switches instantly.