File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes XML and Related Technologies and the fly likes XML generation Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Engineering » XML and Related Technologies
Bookmark "XML generation " Watch "XML generation " New topic
Author

XML generation

sonu arora
Ranch Hand

Joined: Jul 25, 2002
Posts: 30
Hi all

i m using xerces DOM parser to generate a XML file . firstly the plane data ll come from tables of a database and populate the DOM parser memory.
the problem here is that the data in the tables r quite large (in GB's )and parser throw java.lang.OutOfMemoryError exception . the resulting XML file ll be around 500 MB . so a single file is not feasible here. what i want to do is to generate a number of XML files (no single file). what should i do here to draw a line for making another XML file , like 10 no's of main element in a single XML file , size of document in the DOM memory.
i can fix the no's of elements fo one XML file , but how can i fix the size of each XML file here.
plz suggest
try & try u ll surely win.


try & try u ll surely win.
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Would it be possible to skip constructing the XML document in-memory (DOM) but write it directly into disk instead? It's not difficult to write an object-to-XML serializer if you know what kind of objects you'll need to serialize. You could even encapsulate the file switching into that serializer class (as it has access to the output stream and can keep track of actual bytes written). Here's a sketch for the use of such a utility:


Author of Test Driven (2007) and Effective Unit Testing (2013) [Blog] [HowToAskQuestionsOnJavaRanch]
sonu arora
Ranch Hand

Joined: Jul 25, 2002
Posts: 30
Thx for ur reply
it is not possible to skip the DOM here .what is the memory limit use by DOM parser .and can i change it.
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
When you read an XML document into a DOM tree, it's all in memory. Even if you only needed 15% of the actual data. That's where the memory limit shows itself. If you have a 50 megabyte file containing order receipts and you want to print out only the order IDs using DOM, you'll end up with anything between 50-100 megabytes (maybe even more) of memory being used for storing the corresponding DOM tree in memory while 1 megabyte would've sufficed if you used a custom SAX handler for constructing an object hierarchy containing only the order IDs. There's nothing much you can do except buy more memory or lighten the XML document (e.g. by dividing it into several smaller documents) if memory consumption becomes a bottleneck.
In short, DOM is easy but comes with a cost in performance.
sonu arora
Ranch Hand

Joined: Jul 25, 2002
Posts: 30
i m not reading a XML file here . i m generating a XML file using DOM parser.what i want to know is if at any moment ,can i know the size of XML file or not,which would be in DOM memory at that moment.
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
what i want to know is if at any moment ,can i know the size of XML file or not,which would be in DOM memory at that moment.

The only way I know is to serialize the DOM tree into disk and read that file's size (which may or may not be similar to the amount of memory being consumed when holding the DOM tree in memory).
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: XML generation