Number keys not directly switching items in containers

Using number keys to switch items doesn't work correctly in containers

To reproduce:
1. place chest and put in items
2. hold an item in your hot bar
3. hover over one of the items in the chest
4. press the number where the item is in your hot bar (eg. slot 1 press "1")

The items will not be switched as they should be but instead both moved into the hotbar