Sorting a large CSV file

[Tom Roffe]

hello all,
just a quick one, this is my first post to the forum and i like what im seeing ;-) . Any way i have a bit of a problem. i have a 180Mb CSV file and i need to sort it in name and post code (ZIP) order. i have a small app to sort it and it works fine on small files, but when i try to sort the beast of a CSV it gives me

java.lang.OutOfMemoryError
Exception in thread "main" 

i also have a smaller CSV around 20Mb and that gives the same error. does any one have some suggestions if so they would be greatly appreciated.
CODE BELOW
thanks in advance..
Tom Roffe

import java.io.*;
import java.util.*;
class FileSort2 
{
public static void main(String[] args) 
{
      PrintWriter outFile;
      BufferedReader inFile;
      ArrayList names = new ArrayList();
      try{
  inFile = new BufferedReader(new FileReader("PcustSM.CSV"));
  String line;
  while( ( line = inFile.readLine() ) != null) 
{
names.add(line);
}
  inFile.close();
}
      catch (IOException e) 
{
  System.err.println("File error: " + e.toString() );
  System.exit(1);
}
      Collections.sort(names, new Compare());
  //ShearSortAlgorithm.sort(names);
      try{
 outFile = new PrintWriter(new FileWriter("Surname_Sort.txt"), true);
 for (int i = 0; i < names.size(); i++)
outFile.println((String)names.get(i));
 outFile.close();
}
      catch (IOException e)  
{
 System.err.println("File Error: " + e.toString() );
 System.exit(1);
}
}
}
class Compare implements Comparator 
{
   public int compare(Object one, Object two) 
 {
  String first = (String) one;
  String second = (String) two;
  String f1 = getForename(first);
  String f2 = getForename(second);
  int f = f1.compareTo(f2);
  if ( f != 0) return f; 

  String s1 = getSurname(first);
  String s2 = getSurname(second);
  return s1.compareTo(s2);
}
   private String getForename(String name) 
{
  String[] part = name.split(",");
  return part[1].trim();
}
   private String getSurname(String name) 
    {
  String[] part = name.split(",");
  return part[0].trim();
}
}

[Dmitry Melnik]

Hi Tom. Here are a few things to try:
1. Run the virtual machine with this parameter "-Xmx1500m" It will set the memory size for VM, the default one is too small for your task.
2. Optimize the memory usage of your task. For instance, instead of inserting into the list raw strings read from the input file, insert arrays of strings (the results of split), and update your comparator and result file output code accordingly.
3. You might want to sort your data by inserting helper objects representing your input strings into sorted collection (like TreeSet). You will need to take care of duplicate keys though.

[Tom Roffe]

BIG Thanks Dmitry Melnik, worked treat.
one other problem has just presented it's ugly self. The input file has some entries that are in CAPS and others arn't. When the program sorts the list the CAP'ized entries are at the top of the list sorted and the non-caps entrys are sorted but at the EOF.
Question, how can i make the sort process ingore the case of the file entries.

example of output file
AbdiShalim
AbdiSwanson
...
...
ZiaNyoni
ZiaXu
ZubairAvaujla
ZubairMoran
aaronAyadcock
aaronKashem
abbeyJanger
...
...

[Dmitry Melnik]

Before you start sorting convert to the same case (with toLower(), toUpper()) the strings you compare.