I have a requirement to create a web service client to upload 150,000+ records from our database. My main question is whether others are creating web services to process bulk data and how they handle memory concerns.
I began with the standard API, populating the jax-b annotated beans created from the wsimport. I used JConsole to analyze memory usage. I noticed that there is a sharp spike in the plot of the heap usage (from 200MB to 800MB) right after the invoke method call.
The problem is that the max heap size on our app server is set to 1GB and since this application shares it with other applications I keep getting a out of memory error.
I was asked to look for alternatives, so I tried using the Dispatch API creating the payload by sub-tree marshaling to StAX. The issue I then ran into is that I have a outputstream and the dispatch api is expecting an inputstream. I tried the options found here http://ostermiller.org/convert_java_outputstream_inputstream.html
However after using jconsole to analyze again it appears the writing to a temp file or using a byte array increases the amount of memory being used to almost equivalent of using the standard API. I tried using piped streams in separate threads and the single threaded circular buffer methods, but the application seems to hang after the invoke method call.
Richard Adrian wrote:
Unfortunately for this I am only writing the client and the service expects everything in a single request. They are truncating their tables and replacing with every request.
I understand your pain. As far as I understand, there is not much you can do, as long as the design of the service is unchangeable.
If you could modify the service, then some options are:
- Operating on a subset of the total amount of records.
- Transferring data from the database in an attachment.
- Consider a REST web service that uses JSON.
Note that these are just guesses, since I do not know about the detailed requirements.
Best wishes and sorry I cannot help you more.