The server I run on my pc has a specialized issue.

The problem my server has is that once I start the server, if my friends do not immediately the server it won't let them on at all. The only way they would be able to get on is if I restart the server launcher. My server at one point used to not have this problem. The error report people receive when they join is "java.net.ConnectException: Connection Timed Out: no further information". This problem is server side.
From this would you be able to tell what the problem is, minecraft or non-minecraft related? I've had this problem for over a year across all the updates.