Very Slow and Laggy Chunk Loading for 1.6/.1

I was flying around on a new world and looking around when I noticed the chunk loading took a really really long time to load and was causing some lag. After I flew through and loaded them it seemed fine. But I had to actually enter the chunks for them to load. And once when i entered and unloaded chunk it didn't load and I started slowly floating down. Then after a minute or so it loaded.