permaculture playing cards*
The moose likes Web Services and the fly likes Sending big file using web service Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of OCA/OCP Java SE 7 Programmer I & II Study Guide this week in the OCPJP forum!
JavaRanch » Java Forums » Java » Web Services
Bookmark "Sending big file using web service" Watch "Sending big file using web service" New topic
Author

Sending big file using web service

Giang Truong
Greenhorn

Joined: Sep 16, 2011
Posts: 4
I have to send a big base64 file to a vendor. I use the wsdl2java tool to convert vendor's WSDL to Java stub classes and call a web service method to upload the file. The file content is inserted into the SOAP message as a member of a complex type.

When the file is big, 10MB or so, I run out heap space.
Normally, using MTOM will solve this kind of problem, however, MTOM requires that both client and server implement MTOM. In my case, the vendor will not implement MTOM.

Is there any alternatives that I can send the file? I am thinking about streaming the SOAP message, so instead of loading the whole file into the message, I will write one chunk at a time till I complete the file. IS THIS POSSIBLE at all?

Thanks,
Raja Pal
Ranch Hand

Joined: Jul 12, 2004
Posts: 92
Just being a little pragmatic, did you evaluate that this situation actually needs a web-service - I mean the overhead of SOAP and schema typing a complex type, etc. Have you explored a simple change of options with the method invocation/messages still being sent the SOAP way(wsdl) but the file(and I have a hunch this is just some serialized data stream) can be sent using good old FTP, which is most cases can be supported by the vendor end of the service channel? If this is ruled out, then you may consider breaking the SOAP packet size to less granular chunks.


Java Pal - Your friend in technology and innovation...India.
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12809
    
    5
SOAP encoding large objects is a really bad idea - thats why SAAJ SOAP API for Attachements in Java was invented.

The overhead of SOAP encoding and decoding can be avoided by using SAAJ.

As Raja Pal said - maybe you should step back and consider a simpler solution.

Bill
Giang Truong
Greenhorn

Joined: Sep 16, 2011
Posts: 4
Raja, William:

Thanks for your response.
As Raja suggested, there are other ways to achieve the goal like ftp, https, but web service does have some extra benefits.
I try to push Web Service as much as I can but it looks like I have reached the limit.

William:
Can you elaborate a bit more about the SAAJ? Do you have any code skeleton that I can follow?

Thanks,


William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12809
    
    5
Can you elaborate a bit more about the SAAJ? Do you have any code skeleton that I can follow?


No, the link I cited should get you started. Use Google to find SAAJ examples for whichever SOAP toolkit you are using.

Bill
Ivan Krizsan
Ranch Hand

Joined: Oct 04, 2006
Posts: 2198
    
    1
Hi!
Section 5.7 of my electronic book (freely downloadable here http://www.slideshare.net/krizsan/scdjws-5-study-notes-3085287) shows a complete example on using SAAJ.
Section 10.3 contains a complete example (client and service) showing how to use MTOM.
If I recall correctly, it automatically handles placing the data in an attachment. I may be wrong, though.
Disclaimer: All examples uses the Metro JAX-WS reference implementation.

I recall having read an article about being able to stream very large attachments with Metro and SOAP web services, but I cannot seem to find it at the moment.
Best wishes!


My free books and tutorials: http://www.slideshare.net/krizsan
Giang Truong
Greenhorn

Joined: Sep 16, 2011
Posts: 4
Ivan's doc is incredible. Lots of details and explanations.

Here is what I have found. Correct me if I make any mistake.
I changed my code to use SAAJ, and the setContent() helps a lot. Normally, in order to create the SOAP message, I had to maintain 2 copies of a string: one is the Base64 string representing the file, and the other is the Base64 string of the file inside the SOAP message.

If I send a 20Mb word doc, it is equivalent to 26Mb Base64 string. In order to keep one copy of the 26Mb Base64 string, the heap space grows to around 120Mb.
2 copies will take up 240Mb heap space easily. The setContent() maintains only one copy of the string in form of byte[] and thus reducing the heap space to only 120Mb.

However, 120Mb heap space is till a lot.

The word streaming in SAAJ gives me the impression that I don't have to keep a full copy of the SOAP message in memory and that I can keep the SOAP message in a FileInputStream (or something similar) and send the content of a stream one chunk at a time. So far, it is not the case.

Thanks


Giang Truong
Greenhorn

Joined: Sep 16, 2011
Posts: 4
I just found this piece of code

http://www.java-tips.org/other-api-tips/httpclient/how-to-send-an-xml-document-to-a-remote-web-server-using-http-7.html

In the strURL is the URL of the web service

This block of code will solve the heap memory issue



And, the Post Method getResponseBodyAsString() contains the web service response.

This is so simple.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Sending big file using web service