aspose file tools*
The moose likes XML and Related Technologies and the fly likes Reading Very Big XML Files Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Engineering » XML and Related Technologies
Bookmark "Reading Very Big XML Files" Watch "Reading Very Big XML Files" New topic
Author

Reading Very Big XML Files

pradeep selvaraj
Ranch Hand

Joined: Nov 29, 2005
Posts: 62
Hi we are currently using JDom to read xml files. The file range from 10MB to 40MB. Most of the smaller ones (the 10Mb files) are read and populated in our applicaction at about 15 seconds. As the file size gets bigger, it takes unacceptable amount of time for the users and sometimes the application even hangs. Are there any other APIs that are good for reading big xml files, quickly?

Thanks in advance

Pradeep


[<a href="http://cyvis.sourceforge.net" target="_blank" rel="nofollow">CyVis - Software Complexity Visualiser</a>] [<a href="http://pradeepselvaraj.blogspot.com" target="_blank" rel="nofollow">Pradeep's Blog</a>]
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12803
    
    5
The key point you have not said anything about is what do you do with the file when it has been read in? If you have to manipulate the resulting DOM then why not use the DOM parser built in to Java since version 1.4?

Bill
pradeep selvaraj
Ranch Hand

Joined: Nov 29, 2005
Posts: 62
Yes we do manipulate the data, but we feel that its a bit slower than we would like it. So are there any other tools available, which you think might be useful to read and manipulate very big xml files?

Thanks
Pradeep
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12803
    
    5
In the early days of XML there was a lot of interest in "lazy" parsers which would make a pass through a document locating the main elements but leave lower parts of the hierarchy in plain text form unless needed. I just did a Google search for "lazy xml parser" but most of the references were old.

Lately there has been interest in something called "Fast Infoset" - a loss-less binary representation of a XML document in a more easily parsed form. See for example, this page in the Glassfish project. Whether this is applicable to your problem, I don't know.

Bill
 
It is sorta covered in the JavaRanch Style Guide.
 
subject: Reading Very Big XML Files