Posting large files using HttpURLConnection

[Michael Bronshteyn]

I am trying to post very large file to the web server using the following code:
// get file to copy to server
in = new FileInputStream( contentFile );
// get HttpURLConnection
conn = ( HttpURLConnection )m_url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setUseCaches( false );
// set request propertieis
setRequestProperties( conn, requestProps );
conn.connect();
OutputStream out = conn.getOutputStream();
// setup loop to read content of the file
// and write to the output stream
byte[] data = new byte[ 4096 ];
int bytesRead = 0;
while(( bytesRead = in.read( data )) != -1 ){
out.write(data, 0, bytesRead );
out.flush();
}
out.close();
retStatus = conn.getResponseCode();
The problem is that this code attempts to read entire file into memory and than sends it to the server. Is it a way to send file by reading a chunk into memory and sending it over through OutputStream?
Thanks a lot.

[Peter den Haan]

What makes you think the entire file is read in memory? As far as I can make out, the HttpURLConnection uses a plain BufferedOutputStream wrapped around the socket's OutputStream under the hood.
- Peter

[Michael Bronshteyn]

Hi Peter.
Thanks a lot for your reply.
I have established connection with remote debugger to the remote servlet. The breakpoint has been set in doPost method. I have another session with the debugger in the code which posts using HttpURLConnection. I can see that remote sites trips the breakpoint only after I am done with the loop and call "getResponseCode" method.
The reason I noticed this problem I am trying to post 90M file to remote site. While this code works fine for smaller files, this file causes "OutOfMemoryException". I have ran the code which posts using hprof switch ( -Xrunhprof ) and it showed a lot of heap allocations in the traces reading file in the loop.

[Peter den Haan]

Yikes. What JDK are you using? The JDK version I (don't tell anyone) decompiled before posting my reply is 1.4.1 -- did not do a full analysis though, so there can still be a problem hiding somewhere.
You could always try the Apache Jakarta HttpClient library. At least it's open source and you can mend things if they break
- Peter

[Michael Bronshteyn]

Hi Peter,
I am using jdk 1.3.l ( one of the system requirements ). Thanks a lot for suggestion to use Jakarta library. I am going to give it try.
Thanks a lot,
Mike.

[Raj Kethu]

Hi Mike,

Have you got the solution for this. I am also facing exactly the same probelm and its very important part of our application. If you found the solution. please paste the code or ideas here.

Thanks,
Raj

[Raj Kethu]

I investigated on this problem and found the solution.

Problem:
I have a Huge file that should goto server but, that should go chunk by chunk not at once(because, the file is 250 mb in size, i dont want to store that in memory). I used default JDK java.io and nio but, it did not flush to the server until I close the stream (or you get the 200 (OK) from the server.

Solution:
The default JDK stream does not flush to servlets (or server programme) even if you use stream.flush in your programme. But, if you use the following apis you can overcome this problem of flushing.

1) Servlets - Servlet clent can flush every chunk everytime.
2) HTTPClient - It works same as servlets but, in case if you are not using servlets (In my case, i cant use servlets because, I should not contact webserver). This works same as servlets.

Hope this helps to someone.

[Michael Bronshteyn]

Using Commons HttpClient solved the problem. Thus you can try the same approach. It should work for you.