Pick block when holding an enchanted item replaces wrong item

When I try to replace an enchanted item in my hotbar using pick block instead of replacing that item it will instead replace a different item. Basically its impossible to replace them using pick block. Very annoying!