Experience points keep lowering

My first time playing here. I have been enjoying this game more than I anticipated yet I’ve run into a slight problem.  At some point when I started figuring out what to do in my first world I had run into mercenaries with a banner above one of them.  After taking them out I was informed that I caused a bad omen feature to occur and noticed within a couple minutes that my XP (levelling up bar) was decreasing significantly.  

I've attempted to google search how to rid this because I just hit level 21 and dropped to 16 on the first decrease, than 13.  I drank the milk sometime after and was at level 10 but today I realized I was still going down even after the bad omen duration was over (100 minutes).  

I'm told levelling up isn’t that much of a thing you need unless you’re enchanting yet I still find it bothersome that I could be level 30 and I’m stuck dropping down to level 1 over and over ever. 

can this be fixed??