Serious problem happening Minecraft Realms server.

Normally, it takes only few seconds to log in to server. But sometimes it takes more than a minute or over. And when that happens, it usually comes up with 'banned by administrator' message and literally kicked out from server both me and my friend. Once we kicked out from server, and we try to get in to play again(of course it takes minutes to log in to server), the message pops out like every five seconds, which is just unable to play in my Realms server with my friend. Plus, sometimes server rolls back.

When this problem happens, it's unable to change server's settings. Like closeing server, changing the world, and invite friend to the server.

This problem is happening too often that makes me and my friend soo diturbing to play in minecraft realms server.