Reading and Modifying a Big CSV file

[Debashish Chakrabarty]

I have a task where there are a lot of bzip files containing big CSV files (to the tune of 50MB or more, when uncompressed). I have to uncompress them, parse them, change some of the tokens with new text taken from an excel sheet, compress it again.

Now I am calling Runtime process call for Zipping/Unzipping part and am doing it alright. I have problem with the following code that reads the CSV and makes changes to it. When run normally it exits due to an OutOfMemoryError. I then ran it as java -Xms50m -Xmx100m com.mypkg.utility.CSVReader. It works, but is very slow.

Few questions hence,
(1) How can I manage it without increasing the heap size?
(2) How can the following code be improved?
(3) Is storing the stuff in a StringBuffer and then writing it in a single go a good approach?

Thanks for your time,

Debashish

Here is the code snippet:

BufferedReader reader = new BufferedReader(new FileReader(csvFileName));
int lineCount = 1;
String DELIMITER = ",";
 
//Iterate through all lines
for(String line = null; (line = reader.readLine()) != null; lineCount++) {
    int columnCount = 1;
    
    //Iterate through all tokens in the line
    for(StringTokenizer tokenizer = new StringTokenizer(line, DELIMITER);
 tokenizer.hasMoreTokens(); columnCount++) {
        Object objElement =  tokenizer.nextElement();
        String strToPrint = (String) objElement;
        //System.out.println("[" + lineCount + DELIMITER + columnCount + "] = " + strToPrint);
        
        //Some processing to replace an existing tone with some new text
        if(conditionSucceeds){
            strToPrint =  "NewText";
            sbResult.append(strToPrint);
        }
        else{
            sbResult.append(strToPrint);
        }
        
        //We need to take care if the string itself has a delimiter (we know it
        //with the sorrounding quotes ""). If it does append the next token as well.
        if((strToPrint.indexOf("\"") == 0) && hasDelimiter(strToPrint)) {
            //System.out.println("This string has delimiter at " + strToPrint.indexOf(DELIMITER));
            objElement =  tokenizer.nextElement();
            strToPrint = (String) objElement;
            sbResult.append(DELIMITER).append(strToPrint);
        }
        if(columnCount != LASTCOLUMN) sbResult.append(DELIMITER);
    }
    //Now append a new line char
    sbResult.append(LN);
}
String strTemp = sbResult.toString();
strTemp.substring(0, strTemp.lastIndexOf(DELIMITER));
 
//Write the CSV to a temporary location
writeToFile(strTemp, strOutFile);

[ April 13, 2005: Message edited by: Debashish Chakrabarty ]

[Ben Wood]

It will be much quicker if you do something like...

 
 
loop
 {
 read next line into string
 parse line and make edits required
 write line to temp file
 }
 
 

...also perhaps it would be quicker still if you read a few lines, say 10 for example, into a StringBuffer and then wrote those 10 back once edited to the temp file, rather than doing an IO write() for each line. So, perhaps the loop will contain a test condition to see if 10 lines have been read, and if so write the current contents of the StringBuffer to the temp file and then clear the StringBuffer and reset a 10 line counter variable.

[Sripathi Krishnamurthy]

If are using JDK 1.4 and above, you can use the replace function

read the line from the file
use the replace function //replaceAll("oldstring","newstring")

Also with my prior experience with file handling, you have to give an increased heap size for JVM for this.