• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Download parts of file separately

 
Ranch Hand
Posts: 94
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

I'm trying to develop a download manager using java. I would like to make it multithreaded. Now, I would like to know how to download different parts of the file in separate threads.

Example,

If file size is 50M then it must be downloaded by 10 different threads, 10M in each. Is using the skip() method the only way or is there a better way of doing it?
 
Bartender
Posts: 9626
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Any chance you know Valery ? He was asking the same thing last week.
 
Chandra Sekar
Ranch Hand
Posts: 94
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I want it not only for FTP but also for HTTP. Also, the answers in that post does not make anything clear to me.

Please bare with me and explain it in simpler terms. Also, please tell me is using the skip() method cause overhead by reading and skipping the bytes.
 
Joe Ess
Bartender
Posts: 9626
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Stan asks a very good question in that thread: " What bottleneck constrains one thread that will not constrain two or ten?" Why are you trying to download a single file with multiple threads when the bottleneck is likely network throughput, which adding additional threads will only make worse?
And yes, skip() reads in bytes in order to "skip" them. RandomAccessFile has the ability to move a file pointer around without reading bytes.
 
Chandra Sekar
Ranch Hand
Posts: 94
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
In my project I'd be having download servers, where the various parts of the file would actually be downloaded in separate systems. The client sends the starting and ending byte locations and the other system would download only that part of the file.

How do I handle this scenario?
 
Chandra Sekar
Ranch Hand
Posts: 94
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Can anyone please help me out of this?
 
(instanceof Sidekick)
Posts: 8791
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Looks like HTTP supports byte ranges (see 14.35.1) You'll have to hope that the servers you hit actually implemented them.

Your client would have to set the proper headers before requesting data. If you haven't built such requests before, look into the Apache Commons HTTPClient package.

This sounds like a scary architecture ... you have to be certain that each of the many servers you hit have exactly the same version of the file, right?
 
Ranch Hand
Posts: 547
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
sounds like P2P filesharing ?
 
Chandra Sekar
Ranch Hand
Posts: 94
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The various parts are downloaded from the same remote server but by different systems in the LAN.

It works like,

First 100 KB by system 1;
Second by system 2 and so on.
 
Chandra Sekar
Ranch Hand
Posts: 94
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Please bear with me and help me dear ranchers!!!

 
Chandra Sekar
Ranch Hand
Posts: 94
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I got it solved!!
 
Stan James
(instanceof Sidekick)
Posts: 8791
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

The various parts are downloaded from the same remote server but by different systems in the LAN.



Oh, I had that part backwards. So how did you make it work? And what do these multiple downloaders do with their separate parts?
 
reply
    Bookmark Topic Watch Topic
  • New Topic