• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Reading a max number of files from a dir

 
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello,

I have a directory with a large number of files (several 10,000 files). I want to process them chunk-wise and therefore want to read them in chunk by chunk. However, when I use File.list(...) or from Apache Commons FileUtils.iterateFiles(...) the respective function iterates over all files also when I have collected the files for my next chunk already which is a waste of CPU time. When my chunk size has been reached I simply return straight away false from within the FileFileter.accept method. Nevertheless, this does not prevent from all the remaining files I'm not interested in any more for the current chunk to be iterated over. And if the number if Files in the dir is large this may take some time...

Does anybody know of some library hat will stop iterating over the files in a directory when some max value has been reached? I couldn't find anything and spent quite some time surfing the Internet.

Thanks, Oliver
 
Greenhorn
Posts: 2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Maybe it's worth looking into Spring Batch ? There might be a learning curve involved though :P
 
Marshal
Posts: 79177
377
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Welcome to the Ranch both of you.

Doesn’t the File class return the contents of a directory as an array? In which case, you can iterate part of an array with a for loop (not for‑each).
 
Sheriff
Posts: 22783
131
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If you're using Java 7, using a DirectoryStream may be an alternative. It (probably) doesn't retrieve all of the files and sub folders into an array, so it may be a bit more efficient. Just use the following example:
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic