Win a copy of Re-engineering Legacy Software this week in the Refactoring forum
or Docker in Action in the Cloud/Virtualization forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Sorting huge files by using an index

 
Prashant Sehgal
Ranch Hand
Posts: 56
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

How does one index a flat (CSV) file for the purposes of sorting? If I know of the exact columns on which I have to sort (or generate the index for), what are the various techniques available for doing it? I read through the web and all indexing related tutorials were targeted towards DBMSes. Does anyone know of a tutorial for indexing flat CSV files? Or has anyone reading this post tried it before?

My file is not fixed width, so using a RAF is not recommended.

I know of internal and external sorting algorithms, but have never tried indexing a flat file before. How does it work?

Thanks,
Prashant.
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Can you please explain what the purpose of indexing the file is? How will the index be used?
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13056
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Since your record length is not fixed you are going to end up using a RandomAccessFile to make use of the index - it is the only way to jump to a record start.
Off the top of my head I would say that you will have to scan the file looking for the starts of lines, for each line start record the file position and grab the content you are going to sort on - sounds like a job for a custom object containing two variables:

long fposition ;
String key ;

store those guys in a collection - maybe a TreeMap - when your scan is done, the TreeMap will have the sorted order and the fposition will point to the line start so you can do a RAF seek to it.

Bill
 
Prashant Sehgal
Ranch Hand
Posts: 56
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Has anyone heaad of Lucene?

It's an indexing API form jakarta.org
 
Richard Rodger
Greenhorn
Posts: 17
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How big is the file? Simplest solution is just to read in all in, sort in memory and dump it out again.

Otherwise go with a simple in-process database like http://hsqldb.sourceforge.net
and load the CSV file into a table
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic