• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Sending a large XML file and an ArrayList

 
Ranch Hand
Posts: 61
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi everybody,

I am currently designing a system in which large xml files (approximately 1 Gb) are parsing by SAX and values in it are filled a huge cache. I decided to seperate parsing process and caching process into two machines. In the first machine i am parsing the document and filling the values (as Strings) into an arrayList (i am not sure this is the best collection type).
My question is about the transfer of this arraylist to other node, my cache. These two nodes are in local network, what is the best way to send this huge arraylist to other node? socket communication over TCP, RMI, JMS or other alternatives? Performance, round-trip-time is important for me and parsing interval of XML files is 24 hours.

Could you make any suggestion?
Thanks lot,

Serkan
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
1. You should probably think in terms of transmitting a String[] rather than an ArrayList.
2. Why not just serialize the String[] to some shared file location and notify the other application with JMS. That way communication is asynchronous and would still work if the other application is temporarily busy or offline. If you don't delete the previous serialized String[] until the new one has been written, you remove a possible failure point.
Bill
 
Ranch Hand
Posts: 776
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I agree that shared disk is the best way to handle this. Notification can be made my JMS or even cruder methods: create the output file, then create an empty trigger file.

The other side just watches for the trigger file, then processes the actual data file (with a trigger file delete of course).

If you must go with Network transfer, basic Socket IO will get you the best performance.

Guy
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic