I've come so far with this project I can't bear to give up on it now, there's just one little snag that's left in my path and I can't figure my own way around it. If anyone could please help me with examples or point me to some, that would be fantastic!
I'm making a web service with the main purpose of sending and receiving extremely large image files (high resolution telescopic pictures of galaxies, etc.). It's very important that this service be interoperable, so as such I've been confined to using DIME formatted attachments since both Axis and .NET supports them.
Where I'm at now: The service is running in .NET and is capable of both sending and receiving the large image files. As some of you may or may not know, DIME has a built in limit of 4096KB. It's easy enough to override this on the server side by just modifying the web.config and throwing in whatever size you want. The same goes with a .NET client, which I also have running and is capable of sending and receiving the large image files. Again, I just modified the app.config with a bigger number. The problem arises with the Java client, which, oddly enough, seems perfectly capable of receiving the large image files, but unable to send them. Whenever I try I am just presented with a 400 Bad Request error.
Where I need to be: Ideally we'd like to have the service running on a Linux server since we're already well equipped to handle that. Mono is out of the question because it doesn't have support for WSE2 and therefore, no attachments. Therefore I need to get a Java service going, but I need a way to get around the 4096KB limit. Otherwise, pretty much all I'll be able to do is have the Java service receive files from the C# client since it can't send any out or receive from the Java client which will also be incapable of sending.
Perhaps that was a bit more than you needed to know, but here's The Big Question How can I send DIME attachments from Axis that are greater than 4096KB? Is it somehow, somewhere possible to override this built in max length? I've seen things about chunking, but that presents two more problems 1) I don't know how to do it (can anyone give examples or point me to a tutorial?) and 2) it's my understanding that all that chunking does is break up a file into several small attachments and append them to the same SOAP message. If that's the case, the 4096KB limit is applicable to the entire message, SOAP and all attachments, so it'd be no good.
As an alternative, we've been considering MTOM. Our main concern with that though is that MTOM requires Axis 2.0 and .NET 2.0 with WSE3. Our target audience is very limited (pretty much other astronomers who are interested in programming their own clients to utilize our service) and the higher we set the standards for use, the more we may scare them away. Afterall, Axis 2.0 has only formally been out not even a month or so now I think. But all that aside, are there many out there that feel it would be much better to just go ahead with MTOM for simplicity's sake and the fact that it's the new and upcoming standard?
Joined: Apr 25, 2004
Hi Austen, I doubt I am much further ahead than you, but surprisingly, I need to do exactly the same thing (send large assets via DIME from a .NET client to a Java hosted web service, using Axis 1.2).
So, we got to the point of sending files up to 75Mb reliably, over https, to our web services. We use DIME on .NET clients using Framework 1.1 and WSE 2.0. But, we could do a lot better. I apologize for the point-form below, but here's what little I know:
1. Chunking is used to help your machine's memory requirements, not to send the actual asset. The problem with large assets is, 'most' implementations, including .NET, attempt to read in the entire asset into a buffer (perhaps a byte array) and then attach it to the SOAP envelope. Needless to say, ultra-large assets like your astronomical pics are going to be huge files.
2. The 4096Kb is the default chunk size, you can try changing it later. (So if I am right, the buffer size for reading the file chunks is 4096Kb, or 4Mb).
3. DIME supports attachments up to 2^32 = 4Gb. But, and here is the rub, you can't hold a 4Gb file in any memory all at once. (Even though some Operating Systems allow memory to address a space larger than 4Gb, it'll never work). So, you need to chunk it. That means, reading chunks of the file at a time into memory and slowly attaching each piece.
4. You'll need to edit your App.config to change the maxrequest length, to something like this: ... <microsoft.web.services2> <messaging> <maxRequestLength>75000</maxRequestLength> <executionTimeout value="-1" /> </messaging> </microsoft.web.services2> ... The maxRequestLength is in units of Kb, I believe. You might want to play with this value, too. The file size cannot exceed this maxRequestLength. So you need to arbitrarily set a limit on file sizes.
7. We also will attempt to move to XOP/MTOM one day. We have not yet tried it and so cannot comment on its performance or veracity. We have not yet tried Axis 2.0, but XFire is another SOAP engine to look at for advanced stuff like this.
8. If you try very large assets (>100Mb), you'll run into memory issues. You need to chunk to stop holding the entire file in memory all at once. To reach ultra large files (>4Gb) in theory, you'd have to leave your timeouts way open on your client and JSE, and also find a way to split the file into different subfiles, send each one in separate web service calls and reassemble them at the other end. I would suggest using SOAP headers to uniquely identify each part, and write the parts to disk. Then assemble the parts outside of all of the web service calls, in a separate thread. This will be difficult.
With the above efforts, you should be able to transfer files up to some limit depending on the RAM of your machine. (I have 2Gb of RAM, and can do 120Mb files easily, even though in production officially we stop at 75Mb). I am also restricted when calculating the SSL hash, for memory.