Only able to join when no one else is on

I am the owner of the realm I am attempting to connect to. Whenever anyone else logs on, I usually get disconnected shortly afterwards. Whenever I try to rejoin, it will only go through 1 or 2 phases of the loading screen before saying “unable to connect to server”. I have noticed that I have only been able to connect when no one else was online. I am the only person experiencing this issue. I am able to join other realms just fine.