Haste II does not insta-mine when shift-holding on ladder

I have an Efficiency V pickaxe and a Haste II beacon. And for sure I'm in range.

I just discovered that if I mine stone while shift-holding on a ladder, the haste effect is not applied. It seems to mine at about the normal Efficiency V speed, I think. Regardless, it's far from instamine.

Then if I go ahead and mine a few blocks into a wall at the level I'm holding and walk into that space, the instamine returns.