Sending big file using web service

[Giang Truong]

I have to send a big base64 file to a vendor. I use the wsdl2java tool to convert vendor's WSDL to Java stub classes and call a web service method to upload the file. The file content is inserted into the SOAP message as a member of a complex type.

When the file is big, 10MB or so, I run out heap space.
Normally, using MTOM will solve this kind of problem, however, MTOM requires that both client and server implement MTOM. In my case, the vendor will not implement MTOM.

Is there any alternatives that I can send the file? I am thinking about streaming the SOAP message, so instead of loading the whole file into the message, I will write one chunk at a time till I complete the file. IS THIS POSSIBLE at all?

Thanks,

[Raja Pal]

Just being a little pragmatic, did you evaluate that this situation actually needs a web-service - I mean the overhead of SOAP and schema typing a complex type, etc. Have you explored a simple change of options with the method invocation/messages still being sent the SOAP way(wsdl) but the file(and I have a hunch this is just some serialized data stream) can be sent using good old FTP, which is most cases can be supported by the vendor end of the service channel? If this is ruled out, then you may consider breaking the SOAP packet size to less granular chunks.

[William Brogden]

SOAP encoding large objects is a really bad idea - thats why SAAJ SOAP API for Attachements in Java was invented.

The overhead of SOAP encoding and decoding can be avoided by using SAAJ.

As Raja Pal said - maybe you should step back and consider a simpler solution.

Bill

[Giang Truong]

Raja, William:

Thanks for your response.
As Raja suggested, there are other ways to achieve the goal like ftp, https, but web service does have some extra benefits.
I try to push Web Service as much as I can but it looks like I have reached the limit.

William:
Can you elaborate a bit more about the SAAJ? Do you have any code skeleton that I can follow?

Thanks,

[William Brogden]

Can you elaborate a bit more about the SAAJ? Do you have any code skeleton that I can follow?

No, the link I cited should get you started. Use Google to find SAAJ examples for whichever SOAP toolkit you are using.

Bill

[Ivan Krizsan]

Hi!
Section 5.7 of my electronic book (freely downloadable here http://www.slideshare.net/krizsan/scdjws-5-study-notes-3085287) shows a complete example on using SAAJ.
Section 10.3 contains a complete example (client and service) showing how to use MTOM.
If I recall correctly, it automatically handles placing the data in an attachment. I may be wrong, though.
Disclaimer: All examples uses the Metro JAX-WS reference implementation.

I recall having read an article about being able to stream very large attachments with Metro and SOAP web services, but I cannot seem to find it at the moment.
Best wishes!

[Giang Truong]

Ivan's doc is incredible. Lots of details and explanations.

Here is what I have found. Correct me if I make any mistake.
I changed my code to use SAAJ, and the setContent() helps a lot. Normally, in order to create the SOAP message, I had to maintain 2 copies of a string: one is the Base64 string representing the file, and the other is the Base64 string of the file inside the SOAP message.

If I send a 20Mb word doc, it is equivalent to 26Mb Base64 string. In order to keep one copy of the 26Mb Base64 string, the heap space grows to around 120Mb.
2 copies will take up 240Mb heap space easily. The setContent() maintains only one copy of the string in form of byte[] and thus reducing the heap space to only 120Mb.

However, 120Mb heap space is till a lot.

The word streaming in SAAJ gives me the impression that I don't have to keep a full copy of the SOAP message in memory and that I can keep the SOAP message in a FileInputStream (or something similar) and send the content of a stream one chunk at a time. So far, it is not the case.

Thanks

[Giang Truong]

I just found this piece of code

http://www.java-tips.org/other-api-tips/httpclient/how-to-send-an-xml-document-to-a-remote-web-server-using-http-7.html

In

PostMethod post = new PostMethod(strURL) 

the strURL is the URL of the web service

This block of code will solve the heap memory issue

// Request content will be retrieved directly
// from the input stream
// Per default, the request content needs to be buffered
// in order to determine its length.
// Request body buffering can be avoided when
// content length is explicitly specified
post.setRequestEntity(new InputStreamRequestEntity(new FileInputStream(input), input.length()));

And, the Post Method getResponseBodyAsString() contains the web service response.

This is so simple.