Levitation effect broken with elytra

When I have levitation effect from shulker and I use elytra I can fly down, so the effect doesn't matter anymore. Also by using elytra and having levitation I can fly up and make advancement "Great viev from up here" without effort. Please fix :]