I have a requirement to check the validity of a binary over the internet. Basically I have the database of urls of binaries and their respective size and MD5. I need to frequently check the validity of these binaries (ie. if they still exist in the specified url, and if their size or MD5 has changed). Since I am dealing with large number of URLs, it is taking a lot of time to check all the urls. I am actually downloading each and every binary via java.net.URLConnection and finding out their respective sizes and MD5. Is there any other way you people can suggest to optimize it or help me out to reduce the time of processing.
The bottleneck in a program like that is most likely the speed of your Internet connection. If the program takes too long to run, you should add some log statements to it to see how long the different operations take, or use a profiler to find out where the performance bottleneck is.
But if your Internet connection speed is indeed the limiting factor, then there's nothing you can do about your program; you should just get a faster Internet connection.