FPS drop on difficulty change

For the past few days I decided to go on difficulty hard but after patch 1.6.1 got released the frames dropped rapidly. When I change the difficulty from Easy, Normal or Hard to Peaceful, the fps goes right up to 30 - 150 (w/ optifine and w/o). Changing the difficulty from Peaceful to Easy, Normal, Hard gives me 1 - 7 fps instead of 30 - 150.

Left pic: Difficulty - Peaceful
Right pic: Difficulty - Easy