Extreme lag

I get really high levels of lag on my server, when it happens everyone get the same amount of lag at the same time so its not just me who gets the lag. all of my friends have over 50mbps download and 20mbps upload. My realm name is Neo smp!