Server list shows "Can't connect to server" right away even if server is up

The bug

After loading up minecraft and clicking on the multiplayer button, it shows all the servers in my list, and most if not all 99% of the time come up saying "Can't connect to server" until I click the refresh button.