Win a copy of Rust Web Development this week in the Other Languages forum!

Richard Adrian

Greenhorn
+ Follow
since Jun 04, 2010
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
0
Received in last 30 days
0
Total given
0
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Richard Adrian

Ivan Krizsan wrote:Hi!
Just curious: Do you send all the records in one and the same request or can you consider splitting them up in multiple requests?
Best wishes!



Unfortunately for this I am only writing the client and the service expects everything in a single request. They are truncating their tables and replacing with every request.
11 years ago
I am trying to convert an outputstream to an inputstream in order to construct a StreamSource and came across this article.
http://ostermiller.org/convert_java_outputstream_inputstream.html

My memory requirements eliminate using the byte array or temp file methods mentioned.

I need to create the StreamSource and pass it to another method, so I can't read from the inputstream while the outputstream is being written to. Since I need a buffer all approach, i believe this eliminates the piped streams.

I tried using JBoss' growable pipe streams and the circular buffer, using a join in order to wait until the thread writing to the outputstream completed and then did a flush. I also tried the single threaded circular buffer example.

However when I call the method after passing in the StreamSource constructed with the input stream the application hangs.

Any ideas?

11 years ago
I have a requirement to create a web service client to upload 150,000+ records from our database.  My main question is whether others are creating web services to process bulk data and how they handle memory concerns.

I began with the standard API, populating the jax-b annotated beans created from the wsimport.  I used JConsole to analyze memory usage.  I noticed that there is a sharp spike in the plot of the heap usage (from 200MB to 800MB) right after the invoke method call.

The problem is that the max heap size on our app server is set to 1GB and since this application shares it with other applications I keep getting a out of memory error.
I was asked to look for alternatives, so I tried using the Dispatch API creating the payload by sub-tree marshaling to StAX.  The issue I then ran into is that I have a outputstream and the dispatch api is expecting an inputstream.  I tried the options found here http://ostermiller.org/convert_java_outputstream_inputstream.html

However after using jconsole to analyze again it appears the writing to a temp file or using a byte array increases the amount of memory being used to almost equivalent of using the standard API.  I tried using piped streams in separate threads and the single threaded circular buffer methods, but the application seems to hang after the invoke method call.

Any help would be appreciated.
11 years ago