OOM

[Mary Cole]

Hi,

Am using HttpClient to make a connection to a remote URL to get the file as byte[]. Am getting OOM error when the file size exceeds 10 MB. Is there any alternate way to avoid this OOM

 HttpClient httpclient = new HttpClient();
       GetMethod httpget httpget = new GetMethod(url);
 int status = httpclient.executeMethod(httpget);
            if (status == HttpStatus.SC_OK) {
               byte[] data = httpget.getResponseBody();
               

I think coz of the large amount of data in memory, OOM occurs....any suggestions pls..

Thanks in advance
[ April 25, 2007: Message edited by: Mary Cole ]

[Vlado Zajac]

Don't read all data into memory, if it is not necessary.
(Use getResponseBodyAsStream().)

Or increase maximum heap size for JVM with -Xmx parameter.

[Mary Cole]

Thanks for your reply. Even if I use getResponseBodyAsStream, am getting the same error. Can you pls elaborate on Don't read all data into memory, if it is not necessary

[Jim Yingst]

It looks like the byte[] array "data" contains the entire content of the file. If the file is too big, then that array will be too big. So if there's any way to avoid putting all the data in a single byte[] at one time, do it. For example, you might grab a little bit of the stream at a time, and write the data from the stream to a file:

InputStream in = httpget.getResponseBodyAsStream();
try {
    OutputStream out = new FileOutputStream("outfile.dat");
    try {
        byte[] buffer = new byte[4096];
        for (int n; (n = in.read(buffer)) != -1; )
            out.write(buffer, 0, n);
    } 
    finally { out.close(); }
} 
finally { in.close(); }

Here we still use a byte[] array, but its size is limited to 4096. We just keep filling it and writing it as many times as necessary.

It really depends on what you need to do with the data. Often it's possible to process the data in small chunks, as above. Other times it's much easier to have all the data in memory at once. But for large files, try to avoid putting all the data in memory at once.
[ April 26, 2007: Message edited by: Jim Yingst ]

[Mary Cole]

Thanks for your reply. I got the response using the getResponseBodyAsStream,
now the problem is when I try to insert in the postgres DB, it gives the OOM again. I have written the response in a temp file and using that file as a stream to insert to DB

  Exception in thread "main" java.lang.OutOfMemoryError: Java heap space  

Am seting as -

   ps.setBinaryStream(4, new FileInputStream("outfile_"+ir.getTrackId()+"_"+ir.getTrackKey()+".dat"),ir.getTrackFileSize());
            

can you pls help me whats the alternative soln

[Jim Yingst]

Are you sure this is where the OOME occurs? Either here or when you get to ps.execute(), I guess? If the OOME doesn't have a stack trace (they often don't), you can add logging statements to figure out just where the execution was when the OOME occurs.

If the problem is indeed coming from the PreparedStatement, it seems like this may be a bug or inherent limitation in the SQL driver. You might try checking to see if other drivers are available - a more recent version perhaps? Check the postgresql site for bug reports, or try asking on one of the postgresql mailing lists.

What is the SQL data type of column 4? Perhaps it's something that is not supposed to receive so much data. Though I think that would create a problem on the database side, not on the client side.

Have you tried simply raising the max heap memory with -Xmx as Vlado suggested?

I note that if the goal is to get this data into a PreparedStatement, you can probably skip the intermediate step of writing the data to a file and then reading the file. Just plug the response stream directly into the PreparedStatement:

ps.setBinaryStream(4, httpget.getResponseBodyAsStream());

This probably won't solve your OOME, but it might. And it should be simpler and faster, which is something.

[Jim Yingst]

http://archives.postgresql.org/pgsql-jdbc/2002-08/msg00115.php

Sounds like you might try using setBlob() rather than setBinaryStream().

If you can confirm that you are certain the OOME comes during the postgresql calls, we might move this thread to the JDBC forum for additional ideas.