• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Parsing a growing XML file

 
Greenhorn
Posts: 21
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

I have the need to parse a continuously growing XML file. About once every 30 seconds (although this can variate.. might sometimes be 20, 40, ...), a new record (XML element with sub-elements) will be added to the file and I need to handle that record in my Java program.

Could you give me some pointers as to which parsing API is best to use for this job, and how I can avoid polling for changes and parsing the whole file each time.

Also, the file will not be wellformed since it does not have an end tag for the root element.

Thanks in advance.
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You are going to have to do some tricks before you can present valid XML data to any parser.

I can't see any way around having to poll for changes in the file length or date last modified. I would worry about trying to grab the latest text before the program writing to the file is done.

Once you have managed to grab the most recent extension to the file as a String, it is no problem to add starting and ending elements to create a new String that is a well formed XML document. Which parser to use would then depend on what you actually have to do with the data.

Bill
 
Marshal
Posts: 28193
95
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
All the XML parsers I know of parse complete XML documents only. In fact to be a compliant parser you must operate that way.

So, whoever had this idea didn't take the design of XML into account very well. As Bill said, you are going to have to do some trickery to get anything done. In my opinion it's a really bad idea, and if I had any control over the situation I would throw it out and start over with a better idea.
 
William Brogden
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Actually recording data as XML appended to a growing file is a useful technique for logging complex data from continuing processes.

I use it all the time - you just have to do a little programming before parsing it.

Bill
 
Paul Clapham
Marshal
Posts: 28193
95
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I suppose that's true. And tacking on the end tag form the document element is a simple enough task (you could use a SequenceInputStream so you don't physically have to change the existing document).

But the other part of Jimmy's task is to just process the new bits at the end of the document. Maybe you could do that in some environments with a "tail" command or something like that, but I would just reanalyze the requirements. Whatever is producing the log file should be changed to send a stream of messages over some other channel, and whatever is trying to read the log file should be changed to receive and process these messages. If it's still necessary to have the log file, the latter process could produce the log file.
reply
    Bookmark Topic Watch Topic
  • New Topic