Unable to swing hand on ladder facing up or down (while no entity or block is in focus)

When you look up or down on a ladder you can't attack or use that animation in the air. I f you stare at something it works but it doesn't always happen just like an 80% chance.