aspose file tools*
The moose likes Web Services and the fly likes bulk batch jax-ws web services and memory usage Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Web Services
Bookmark "bulk batch jax-ws web services and memory usage" Watch "bulk batch jax-ws web services and memory usage" New topic
Author

bulk batch jax-ws web services and memory usage

Richard Adrian
Greenhorn

Joined: Jun 04, 2010
Posts: 3
I have a requirement to create a web service client to upload 150,000+ records from our database.  My main question is whether others are creating web services to process bulk data and how they handle memory concerns.

I began with the standard API, populating the jax-b annotated beans created from the wsimport.  I used JConsole to analyze memory usage.  I noticed that there is a sharp spike in the plot of the heap usage (from 200MB to 800MB) right after the invoke method call.

The problem is that the max heap size on our app server is set to 1GB and since this application shares it with other applications I keep getting a out of memory error.
I was asked to look for alternatives, so I tried using the Dispatch API creating the payload by sub-tree marshaling to StAX.  The issue I then ran into is that I have a outputstream and the dispatch api is expecting an inputstream.  I tried the options found here http://ostermiller.org/convert_java_outputstream_inputstream.html

However after using jconsole to analyze again it appears the writing to a temp file or using a byte array increases the amount of memory being used to almost equivalent of using the standard API.  I tried using piped streams in separate threads and the single threaded circular buffer methods, but the application seems to hang after the invoke method call.

Any help would be appreciated.
Ivan Krizsan
Ranch Hand

Joined: Oct 04, 2006
Posts: 2198
    
    1
Hi!
Just curious: Do you send all the records in one and the same request or can you consider splitting them up in multiple requests?
Best wishes!


My free books and tutorials: http://www.slideshare.net/krizsan
Richard Adrian
Greenhorn

Joined: Jun 04, 2010
Posts: 3
Ivan Krizsan wrote:Hi!
Just curious: Do you send all the records in one and the same request or can you consider splitting them up in multiple requests?
Best wishes!


Unfortunately for this I am only writing the client and the service expects everything in a single request. They are truncating their tables and replacing with every request.
Ivan Krizsan
Ranch Hand

Joined: Oct 04, 2006
Posts: 2198
    
    1
Hi!
Richard Adrian wrote:
Unfortunately for this I am only writing the client and the service expects everything in a single request. They are truncating their tables and replacing with every request.

I understand your pain. As far as I understand, there is not much you can do, as long as the design of the service is unchangeable.
If you could modify the service, then some options are:
- Operating on a subset of the total amount of records.
- Transferring data from the database in an attachment.
- Consider a REST web service that uses JSON.

Note that these are just guesses, since I do not know about the detailed requirements.
Best wishes and sorry I cannot help you more.
 
It is sorta covered in the JavaRanch Style Guide.
 
subject: bulk batch jax-ws web services and memory usage