• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Can I read a file chunks in parallel?

 
Maulin Vasavada
Ranch Hand
Posts: 1873
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all,

After long time..

I have huge file (about 500k fixed size records or more). I wanted to know if I can read it in parallel via different threads.

I have not used Java NIO packages yet! (sounds strange, but it is true). Is there a way we can read the file in parallel using any of the NIO concepts? I am still going through tutorial and may be some experiments but I thought of asking here.

Also, if you know of any other/better way of reading a file in parallel other than NIO, it would be great if you share it here.

Regards,
Maulin
 
Norm Radder
Ranch Hand
Posts: 1011
8
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Only if the chunks of the file are on different drives. The OS is going to be the limiting factor on your being able to read different parts of the same file in parallel. In fact trying to do that may slow down the process because the read head on the drive could have to jump back and forth slowing down the process. If you could do the reads at a lower level (ie in machine language) and you knew the relative speed of the disk spinning vs the read time to read a sector from the disk (assuming the disk is sectored), then you could time your reads to start a read a selected sector just before the read head gets to that sector. That might make it possible to read a full track in two rotations (reading sectors 0, 2, 4 and then 1, 3 5) versus reading one sector per rotation. 6 sectors = 6 rotations vs 2.
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13064
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Consider using one Thread to read in records and a separate Thread to process them. You might create a Queue of records waiting to be processed.

I bet you will find that the processing Thread is the limiting factor, not the reading Thread.

Bill
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic