• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Validity of binaries

 
Ranch Hand
Posts: 91
Eclipse IDE Firefox Browser Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi people,

I have a requirement to check the validity of a binary over the internet. Basically I have the database of urls of binaries and their respective size and MD5. I need to frequently check the validity of these binaries (ie. if they still exist in the specified url, and if their size or MD5 has changed). Since I am dealing with large number of URLs, it is taking a lot of time to check all the urls. I am actually downloading each and every binary via java.net.URLConnection and finding out their respective sizes and MD5. Is there any other way you people can suggest to optimize it or help me out to reduce the time of processing.

Thanks
Muthu
 
Java Cowboy
Posts: 16084
88
Android Scala IntelliJ IDE Spring Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The bottleneck in a program like that is most likely the speed of your Internet connection. If the program takes too long to run, you should add some log statements to it to see how long the different operations take, or use a profiler to find out where the performance bottleneck is.

But if your Internet connection speed is indeed the limiting factor, then there's nothing you can do about your program; you should just get a faster Internet connection.
 
Muthukrishnan Manoharan
Ranch Hand
Posts: 91
Eclipse IDE Firefox Browser Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Jesper Young,

I dont have any problem with my internet connection. I could find some improvement in the processing time when I use BufferedInputStream in place of

as I used previously. But I fear BufferedInputStream would have some effects as said here

Or is it alright to go with BufferedInputStream as the problem occurs only with BufferedReader
 
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

But I fear BufferedInputStream would have some effects as said here


What do you mean? That topic talks about the problems of using Readers and Writers for binary data; there's nothing in it about problems using Streams.
 
Muthukrishnan Manoharan
Ranch Hand
Posts: 91
Eclipse IDE Firefox Browser Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Oh.. sorry I misunderstood it..
 
Muthukrishnan Manoharan
Ranch Hand
Posts: 91
Eclipse IDE Firefox Browser Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Will threading be of any use to improve processing time in this application.
 
Ulf Dittmer
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
It sounds as if the processing of each file is independent of that of each other one - so, yes, those could be checked in concurrent threads, likely improving the throughput.
 
Muthukrishnan Manoharan
Ranch Hand
Posts: 91
Eclipse IDE Firefox Browser Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Ulf Dittmer,

For a typical scenario of 43 urls, the application with 5 threads takes little more time than the single threaded version of the application.

Am I going wrong somewhere in the selection of number of threads to process.

Also please tell me which one to choose from the following scenarios:

1. A thread fetches the url, downloads it and then updates in the database. Similarly many threads deal with different urls simultaneously.

or

2. Separate thread for downloading and separate thread for updating (following producer consumer)

I am presently following the scenario no.1.

-Muthu
 
Ulf Dittmer
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Approach #2 makes no sense. It's impossible to say where the problem might be without seeing the code.
 
Just the other day, I was thinking ... about this tiny ad:
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic