Mending does not work if item is not actively held

when having a silk touch pickaxe in the left hand side of the hotbar and a fortune pickaxe in position 2 next to it , the xp will go to the fortune pickaxe untill its full and then will go to the xp bar and never to the silk touch pickaxe (both pickaxes have mending and unbreaking on too)
I have tried without armour and with nothing else in my inventory. i have also tried different positions on the hotbar and all with the same results.