permaculture playing cards*
The moose likes Beginning Java and the fly likes Reading a max number of files from a dir Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Beginning Java
Bookmark "Reading a max number of files from a dir" Watch "Reading a max number of files from a dir" New topic
Author

Reading a max number of files from a dir

Oliver Plow
Greenhorn

Joined: May 18, 2012
Posts: 4
Hello,

I have a directory with a large number of files (several 10,000 files). I want to process them chunk-wise and therefore want to read them in chunk by chunk. However, when I use File.list(...) or from Apache Commons FileUtils.iterateFiles(...) the respective function iterates over all files also when I have collected the files for my next chunk already which is a waste of CPU time. When my chunk size has been reached I simply return straight away false from within the FileFileter.accept method. Nevertheless, this does not prevent from all the remaining files I'm not interested in any more for the current chunk to be iterated over. And if the number if Files in the dir is large this may take some time...

Does anybody know of some library hat will stop iterating over the files in a directory when some max value has been reached? I couldn't find anything and spent quite some time surfing the Internet.

Thanks, Oliver
James Dickson
Greenhorn

Joined: May 18, 2012
Posts: 2
Maybe it's worth looking into Spring Batch ? There might be a learning curve involved though :P
Campbell Ritchie
Sheriff

Joined: Oct 13, 2005
Posts: 36501
    
  16
Welcome to the Ranch both of you.

Doesn’t the File class return the contents of a directory as an array? In which case, you can iterate part of an array with a for loop (not for‑each).
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19543
    
  16

If you're using Java 7, using a DirectoryStream may be an alternative. It (probably) doesn't retrieve all of the files and sub folders into an array, so it may be a bit more efficient. Just use the following example:


SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6
How To Ask Questions How To Answer Questions
 
wood burning stoves
 
subject: Reading a max number of files from a dir
 
Similar Threads
Control number of threads
About the file Delivery
Zipping subdirs along with files
Directory/File comparison tool using Java.
How can I show new text in JTextArea while program is executing?