Elytra randomly stop being activated

When I fly with my elytra I sometimes just fall out of the sky, this is not the durability (or lack there off) because if I tap jump whilst falling I start flying again 

How to reproduce: just start flying

Expected result: you keep flying until you deactivate the wings or land
Observed result: you fall out of the sky