Massive FPS drop when people join world on Xbox Live.

So whenever I start up a world and invite people (or they just join on their own) , my FPS drops to around 1-3 and stays there for like 15 seconds or so. This isn't just me, this issue is happens on all of my friends PC's. My FPS also does the same when somebody enters or leaves the nether. This is probably the worst bug in the beta that I've seen...