Arrows in offhand not prioritized for bow

I have 2  arrows, normal and harming 2, I put normal in offhand so my infinity enchantment works, when firing the bow, it fired my harming 2

Step to reproduce:
1. Get two or more unique types of arrow
2. Put one type in offhand, all of them (if theres 43 speed 2 arrows, then put all 43 in offhand for example)
3. Fire the bow

If it matters, then infinity and normal arrows in offhand