One big XML file or lots of small ones

[Kim Mirtens]

My program uses a lot of static information to do calculations. To make this information more readable to humans I'm migrating from CSV to XML. The CSV files are all separate files for one kind of information.

I've already converted one CSV file to XML and written a test class to retrieve the necessary information. My test class uses DOM and XPath to look up the correct value it needs to do the calculation. I used DOM because I need very frequent access to these values and thought in memory access would speed things up. Is DOM the right way to go or should I use SAX?

Now I also want to convert my other CSV files and put them in the same XML. This would make it more manageable because I would have one file instead of 15. I reckon the total size of this one XML file will be around 2.5 MB (it's all static information so this size won't change). Can I still use DOM in this case? Will XPath be a lot slower when I'm looking through a big DOM tree? Is it better to create 15 seperate XML files and load them as 15 different DOM trees? Will SAX be faster?

[William Brogden]

XPath is kind of slow for repeated lookups. What I do with a DOM of test questions is to create a HashMap where the significant Element references are stored with the question ID as the key. All of the HashMaps are created when the DOM is initially parsed.

This leaves the DOM intact but speeds up the access.

Bill

[Paul Clapham]

Note also that the choice between DOM and SAX is a choice about how you will get the data from external storage into memory, not how you will store the data in memory.

If you don't need the Document that a DOM parser produces, and you just want to produce your own data structure in memory, then you could use a SAX parser. But if you already have something working, I wouldn't rewrite your code on the grounds that your program might start up faster. At this point I would say you have better things to do.

[Kim Mirtens]

Maybe my exact work method will be more clear with my source code. I'm really a newbie when it comes to xml. I'm really uncertain about this code. This is just a test class to access my xml document.

My xml document looks like this:

<?xml version="1.0"?>
<!DOCTYPE hydrostatic [
<!ELEMENT hydrostatic (data+)>
<!ELEMENT data (displacement , draught , lcf , lcb , mct , km , kb)>
<!ELEMENT displacement (#PCDATA)>
<!ELEMENT draught (#PCDATA)>
<!ELEMENT lcf (#PCDATA)>
<!ELEMENT lcb (#PCDATA)>
<!ELEMENT mct (#PCDATA)>
<!ELEMENT km (#PCDATA)>
<!ELEMENT kb (#PCDATA)>
]>
<hydrostatic>
  <data>
    <displacement>22270.300</displacement>
    <draught>4.000</draught>
    <lcf>8.875</lcf>
    <lcb>10.584</lcb>
    <mct>77232.100</mct>
    <km>22.542</km>
    <kb>2.075</kb>
  </data>
  <data>
    <displacement>22864.200</displacement>
    <draught>4.100</draught>
    <lcf>8.786</lcf>
    <lcb>10.539</lcb>
    <mct>77460.400</mct>
    <km>22.096</km>
    <kb>2.126</kb>
  </data>
</hydrostatic>

(of course with a lot more data elements)

My java class to access the data looks like this:

import javax.xml.parsers.*;
import javax.xml.xpath.*;
import org.w3c.dom.*;
import java.io.*; 
 

public class HydrostaticData { 
private Document document;
private XPath xPath;

public static void main(String[] args) {
   HydrostaticData data = new HydrostaticData("hydrostatic.xml");
    System.out.println(data.get("draught","displacement","25000"));
   }

//Build a DOM Document object and initialize the XPath object
HydrostaticData(String file){
try{
DocumentBuilderFactory docfactory = DocumentBuilderFactory 
               .newInstance(); 
         File xmlFile = new File(file); 
         DocumentBuilder builder = docfactory.newDocumentBuilder(); 
         document = builder.parse(xmlFile);

XPathFactory  factory = XPathFactory.newInstance();
xPath = factory.newXPath();

}catch(Exception e){
e.printStackTrace();
}
}
 
/**
 * Method used to get data.
 * @param what String which defines which kind of data we are interested in
 * @param reference String which defines which kind of refenrence data we will use.
 * @param refvalue String that indicates the value of the reference data.
 */
public double get(String what, String reference, String refvalue){
double returnvalue = 0;

try{
// Using XPath get the nodes where the refvalue is in between
// The expression forms a union between the last value of all values lower than the refvalue and
// the first value of all values higher than the refvalue
NodeList nodes = 
    (NodeList) xPath.evaluate(
    "/hydrostatic/data["+reference+" <="+refvalue+"][last()] | /hydrostatic/data["+reference+" >= "+refvalue+"][1]",
    document, XPathConstants.NODESET);

// If the refvalue is exactly a value that is included in the data then 
// there is no need to interpolate and there will only be one node found
if(nodes.getLength()== 1 ){
Element data = (Element) nodes.item(0);
String ret = data.getElementsByTagName(what).item(0).getFirstChild().getNodeValue();
returnvalue = Double.valueOf(ret);
} else {
Element data = (Element) nodes.item(0);
Element data2 = (Element) nodes.item(1);
String refsmall = data.getElementsByTagName(reference).item(0).getFirstChild().getNodeValue();
String reflarge = data2.getElementsByTagName(reference).item(0).getFirstChild().getNodeValue();
String solsmall = data.getElementsByTagName(what).item(0).getFirstChild().getNodeValue();
String sollarge = data2.getElementsByTagName(what).item(0).getFirstChild().getNodeValue();
returnvalue = interpolate(Double.valueOf(refvalue),Double.valueOf(refsmall),
                          Double.valueOf(reflarge),Double.valueOf(solsmall),Double.valueOf(sollarge));
}
 
} catch(Exception e){
e.printStackTrace();
}

return returnvalue;
}
 
/**
 * Method used to interpolate between value
 * refnum is the number for which we want the value.
 * refnum is between refsmall and reflarge 
 * the value (sollution) we are looking for which coresponds to the refnum value is betweeb solsmall and solllarge
 * for instance if we had following table and were interested in what 15 would correspond to
 * 10 -> 1
 * 20 -> 2
 * interpolare(15,10,20,1,2) = 1.5
 */
private static double interpolate(double refnum, double refsmall, double reflarge, double solsmall, double sollarge){
double x = refnum - refsmall;
double y = reflarge - refsmall;
double z = sollarge - solsmall;
double solution = (x/y)*z;
return (solution + solsmall);
}
}

Is this a good way to handle things? Will this give a good performance when I repetitively look up things? What is your advice/suggestions?

[William Brogden]

I worry when I see code like this:

String refsmall = data.getElementsByTagName(reference).item(0).getFirstChild().getNodeValue();    

because it does not provide for the case when getFirstChild() returns null - which is what you would get if the Element is empty. Also if the NodeList is empty you will get a NullPointerException.

How is the performance of your existing code? Have you tested it with a realistic load?

Bill

[Kim Mirtens]

Originally posted by William Brogden:

How is the performance of your existing code? Have you tested it with a realistic load?

Bill

I only wrote this test class. To actually implement it and test it under a realistic load I still need to do some major recoding. I really can't judge performance.
Being a test class it is indeed not error safe. I was more interested in what you thought of my implementation of XML.

[William Brogden]

Having consider what you appear to be trying to do, I conclude that working with the DOM is a bad idea. If this was my problem I would use SAX to parse, and create a POJO (Plain Ole Java Object) for each <data> element.

With a collection of POJO in hand, you are free to make all sorts of lists and Maps of references to the POJO sorted and indexed in various ways that will facillitate the lookup and interpolation. MUCH faster than XPath and other DOM operations and less memory intensive.

Make your POJO serializable and you can write out the whole collection as a serialized list object, but I bet you will find the SAX parsing is pretty fast.

Bill