• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

XML generation

 
sonu arora
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all

i m using xerces DOM parser to generate a XML file . firstly the plane data ll come from tables of a database and populate the DOM parser memory.
the problem here is that the data in the tables r quite large (in GB's )and parser throw java.lang.OutOfMemoryError exception . the resulting XML file ll be around 500 MB . so a single file is not feasible here. what i want to do is to generate a number of XML files (no single file). what should i do here to draw a line for making another XML file , like 10 no's of main element in a single XML file , size of document in the DOM memory.
i can fix the no's of elements fo one XML file , but how can i fix the size of each XML file here.
plz suggest
try & try u ll surely win.
 
Lasse Koskela
author
Sheriff
Posts: 11962
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Would it be possible to skip constructing the XML document in-memory (DOM) but write it directly into disk instead? It's not difficult to write an object-to-XML serializer if you know what kind of objects you'll need to serialize. You could even encapsulate the file switching into that serializer class (as it has access to the output stream and can keep track of actual bytes written). Here's a sketch for the use of such a utility:
 
sonu arora
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thx for ur reply
it is not possible to skip the DOM here .what is the memory limit use by DOM parser .and can i change it.
 
Lasse Koskela
author
Sheriff
Posts: 11962
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
When you read an XML document into a DOM tree, it's all in memory. Even if you only needed 15% of the actual data. That's where the memory limit shows itself. If you have a 50 megabyte file containing order receipts and you want to print out only the order IDs using DOM, you'll end up with anything between 50-100 megabytes (maybe even more) of memory being used for storing the corresponding DOM tree in memory while 1 megabyte would've sufficed if you used a custom SAX handler for constructing an object hierarchy containing only the order IDs. There's nothing much you can do except buy more memory or lighten the XML document (e.g. by dividing it into several smaller documents) if memory consumption becomes a bottleneck.
In short, DOM is easy but comes with a cost in performance.
 
sonu arora
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
i m not reading a XML file here . i m generating a XML file using DOM parser.what i want to know is if at any moment ,can i know the size of XML file or not,which would be in DOM memory at that moment.
 
Lasse Koskela
author
Sheriff
Posts: 11962
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
what i want to know is if at any moment ,can i know the size of XML file or not,which would be in DOM memory at that moment.

The only way I know is to serialize the DOM tree into disk and read that file's size (which may or may not be similar to the amount of memory being consumed when holding the DOM tree in memory).
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic