aspose file tools
The moose likes Java in General and the fly likes Heap size increases when writing XML to the File Big Moose Saloon
  Search | Java FAQ | Recent Topics
Register / Login


Win a copy of The Mikado Method this week in the Agile and other Processes forum!
JavaRanch » Java Forums » Java » Java in General
Reply Bookmark "Heap size increases when writing XML to the File" Watch "Heap size increases when writing XML to the File" New topic
Author

Heap size increases when writing XML to the File

batuk chatuk
Greenhorn

Joined: Dec 29, 2006
Posts: 12
I am converting an object (ArrayList) in XML and then storing that XML as a file on local machine. When there are large number of data, the XML takes too long to write in a File or doesnt write at all (gives some exception).

xmlSource = xmlConverter.convert(object); // object here is no.of ArrayLists..

FileWriter fw = new FileWriter(f);
fw.write(xmlConverter.getXML());

fw.write() takes too much time. Is there any other way which doesnt eat up so much memory ???
David O'Meara
Rancher

Joined: Mar 06, 2001
Posts: 13459

wrap it in a BufferedWriter is the first step

PrintWriter out
= new PrintWriter(new BufferedWriter(new FileWriter("foo.out")));
batuk chatuk
Greenhorn

Joined: Dec 29, 2006
Posts: 12
thanks!!!

File f = new File("foo.xml");
PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter(f)));
pw.write(xmlConverter.getXML(obj));

Using BW does speed up the process by almost 40%, although the PC remains in a numb state during that time..but it does work..

I m afraid this solution is not gonna be enough as there are around 100k entries on Production env at max.. ( I checked above sol. on 50000 records..)

Is there any possibility that i split the records and then in chunks i add them in "same file". (dont want to create more than 1 XML for one page..)

Thanks again...
batuk chatuk
Greenhorn

Joined: Dec 29, 2006
Posts: 12
Its Done!!!

File writing is happening totally with all data at a time in above code..
So this is the answer to better and memory friendly code...

xmlSource = xmlConverter.convert(obj);
BufferedReader bufferedReader = new BufferedReader(xmlSource.getReader());
PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter(f)));
int count = 0;
int buffer=1024;
char[] c1=new char[buffer];
while((count = bufferedReader.read(c1, 0, buffer)) != -1){
pw.write(c1, 0, count);
}
pw.flush();
pw.close();
batuk chatuk
Greenhorn

Joined: Dec 29, 2006
Posts: 12
For 100K records or more the above code didnt work on my machine.. ( i had kept JVM Heap size to Max. 1300K.. )
The reason was the XMLConverter was not able to convert that much data into XML So i wrote the code to divide all the ArrayLists in chunks and then convert them and write them into the same file..

FileWriter fw = new FileWriter(new File("foo.xml , true);

the flag in the constructor is for Appending the new data to same file..
and so file can be generated without any trouble..

Best Regards
 
I agree. Here's the link: http://ej-technologies/jprofiler - if it wasn't for jprofiler, we would need to run our stuff on 16 servers instead of 3.
 
subject: Heap size increases when writing XML to the File
 
Similar Threads
Writing to File
FileWriter Question
FileWriter vs BufferedWriter(FileWriter)
FileWriter
File I/O