Unable to reconnect to server without completely closing minecraft.

I have no issue connecting initially on start of game. But if I quit back to the main menu or server select screen I am unable to connect to the server. I have to completely close the game ad restart it to be able to reconnect to the server.