The moose likes Java in General and the fly likes Can I read a file chunks in parallel? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Java in General
Bookmark "Can I read a file chunks in parallel?" Watch "Can I read a file chunks in parallel?" New topic

Can I read a file chunks in parallel?

Maulin Vasavada
Ranch Hand

Joined: Nov 04, 2001
Posts: 1873
Hi all,

After long time..

I have huge file (about 500k fixed size records or more). I wanted to know if I can read it in parallel via different threads.

I have not used Java NIO packages yet! (sounds strange, but it is true). Is there a way we can read the file in parallel using any of the NIO concepts? I am still going through tutorial and may be some experiments but I thought of asking here.

Also, if you know of any other/better way of reading a file in parallel other than NIO, it would be great if you share it here.

Norm Radder
Ranch Hand

Joined: Aug 10, 2005
Posts: 692
Only if the chunks of the file are on different drives. The OS is going to be the limiting factor on your being able to read different parts of the same file in parallel. In fact trying to do that may slow down the process because the read head on the drive could have to jump back and forth slowing down the process. If you could do the reads at a lower level (ie in machine language) and you knew the relative speed of the disk spinning vs the read time to read a sector from the disk (assuming the disk is sectored), then you could time your reads to start a read a selected sector just before the read head gets to that sector. That might make it possible to read a full track in two rotations (reading sectors 0, 2, 4 and then 1, 3 5) versus reading one sector per rotation. 6 sectors = 6 rotations vs 2.
William Brogden
Author and all-around good cowpoke

Joined: Mar 22, 2000
Posts: 13036
Consider using one Thread to read in records and a separate Thread to process them. You might create a Queue of records waiting to be processed.

I bet you will find that the processing Thread is the limiting factor, not the reading Thread.

I agree. Here's the link:
subject: Can I read a file chunks in parallel?
It's not a secret anymore!