Laggy servers

I assume that this is just because there are less dedicated workers are mojang and more players on realms right now due to COVID-19, but whenever more than 4 people join our realm, the whole thing starts lagging out. Sometimes when there's just 3 people on the server. This also only takes place in the afternoon, morning and late at night, servers are fine.