Lag and RAM usage slowly gets worse in Single Player and Servers for me

I've been playing on my friend's server, and when I join the lag spikes are never too bad, as expected, but if I play for about half an hour, I start getting lag spikes up to 300ms, and it only gets worse from there, with the usage rising and never going back down. Only a few of us on the server have had this problem and I've tried everything from tinkering with RAM allocation to reinstalling the game. I have no idea what is causing the RAM usage to increase.