I have a task where there are a lot of bzip files containing big CSV files (to the tune of 50MB or more, when uncompressed). I have to uncompress them, parse them, change some of the tokens with new text taken from an excel sheet, compress it again.
Now I am calling Runtime process call for Zipping/Unzipping part and am doing it alright. I have problem with the following code that reads the CSV and makes changes to it. When run normally it exits due to an OutOfMemoryError. I then ran it as java -Xms50m -Xmx100m com.mypkg.utility.CSVReader. It works, but is very slow.
Few questions hence, (1) How can I manage it without increasing the heap size? (2) How can the following code be improved? (3) Is storing the stuff in a StringBuffer and then writing it in a single go a good approach?
Thanks for your time,
Here is the code snippet:
[ April 13, 2005: Message edited by: Debashish Chakrabarty ]
SCJP2, SCWCD 1.4
Joined: Aug 14, 2001
It will be much quicker if you do something like...
...also perhaps it would be quicker still if you read a few lines, say 10 for example, into a StringBuffer and then wrote those 10 back once edited to the temp file, rather than doing an IO write() for each line. So, perhaps the loop will contain a test condition to see if 10 lines have been read, and if so write the current contents of the StringBuffer to the temp file and then clear the StringBuffer and reset a 10 line counter variable.
SCJP 1.4, www.gsi3d.org.uk
Joined: Mar 07, 2005
If are using JDK 1.4 and above, you can use the replace function
read the line from the file use the replace function //replaceAll("oldstring","newstring")
Also with my prior experience with file handling, you have to give an increased heap size for JVM for this.