Win a copy of Design for the Mind this week in the Design forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

searching file among 3 hundred thousand files

 
gaurav kumar
Greenhorn
Posts: 16
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all,
i have to search for a particular PDF file among a collection of more than 3 hundred thousand files. I m first making a file object and then taking all the file names into a String array for further processing. But on running this, my server is getting hanged. Is this could be because of very large number of file names in the string array? Could there be some other data structure which can give better performance
thanx
 
Ramen Chatterjee
Ranch Hand
Posts: 62
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Could you be clearer about what you are trying to achieve, ie why are you looking for this file. Also, 300,000 files is a lot! How are these stored? Do you have all 300,000 in one directory (is this possible?).
 
gaurav kumar
Greenhorn
Posts: 16
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hi
the files are reports which get generated periodically. The requirements are in a way that require these many files to be present in a single directory at a time(300,000 is the maximum limit...no of files can also less than this).
Also the piece of code which is giving problem is as below:

File file = new File("D:\\project\\ftp\\Example test invoiced");
String[] filelst = file.list();

thanx in advance
 
Dave Wingate
Ranch Hand
Posts: 262
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
One possible improvement would be to make your directory structure less flat. So instead of having one report directory with 300,000 files, maybe you could group the reports in some meaningful way:



That way, you don't have to create an array with 300,000 members just to iterate through all of the file names.
[ June 23, 2006: Message edited by: Dave Wingate ]
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You might want to index these files as they are created so you don't have to search through every file every time you need something. I use Lucene for indexing with good results.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic