But the only way to find out whether a mail server (or any kind of server for that matter) is active is to try to connect to it. And that unavoidably requires waiting a decent amount of time before you conclude it isn't active. So there isn't any way to avoid waiting for a timeout.
You may imagine some scheme where you have some agent which is continuously
testing to see if a server is active. But that doesn't solve your problem either, since knowing that the server was active two seconds ago tells you nothing about whether it's active right now. And you may find yourself trying to connect with the agent while it's in the process of waiting for a timeout to expire -- what are you going to do then?
So if your actual problem is that having to wait for timeouts is affecting your application's response time in a bad way, then I suggest you redesign your application so that it doesn't connect to the mail server itself. Have it send a message to some other process which is responsible for dealing with your unreliable servers and sending the mail when one is available.