I have a problem where my java code has to read big log files(around 4GB) containing XML and parser those XML to populate data. I can easily parse data using JAXB but i need to know if there is any open source tool provided to read those log files? I have to read these log files to identify a XML request chunk and process it continually till end of file.
I can have java code to read using buffered reader but performance will be an issue then for the reading a 4GB file along with processing data chunk. Can someone please suggest a tool of any design pattern/architectural ideas?
SAX and StAX don't store the entire file in memory, only the separate elements and their text. After it's been processed, SAX / StAX discards it (your code may still keep a reference though). Unless your XML file has elements with millions of attributes, or has enormous pieces of text, SAX and StAX should be able to handle the file just perfectly.