Long connection time with IP instead of hostname

Connecting to a server that's bound to the machine's primary IP connects instantly. However, when connecting to a Minecraft server that is configured to run on the same machine, but on another interface (eg. eth0:1), it can easily take 6 seconds to connect with the IP. If you setup an A record to point to that IP (eg. test.mydomain.com), there's established a connection instantly.

We've tested this with 4 different machines in different datacenters and it's the same for all machines. The Minecraft servers that are bound to the primary ip using ip-address= loads instantly, but all Minecraft servers that are bound to another interface connects slowly unless you're connecting using an A record instead.