Difficulty switching to easy on realms

On my realm, with difficulty set on hard (in the world settings) it keeps randomly switching to easy. This is really annoying. If I want to use some kind of mob farm and forget to type /difficulty hard, it can make it less effective. This just happens randomly, most of the time when there is no one online.