FPS is VERY unreliable in singleplayer

In all gamemodes my fps will hover around 70-90 then without warning drop to 0 or close to 0 for anywhere from 1-60 seconds. Only in singleplayer around once every minute. This is making singleplayer UNPLAYABLE and it is extremely frustrating. PLEASE do something about it! my memory being used never goes above 30% during the low FPS and I always have over 500 mb of memory allocated. It happens in all difficulty settings.