This week's giveaway is in the Android forum.
We're giving away four copies of Android Security Essentials Live Lessons and have Godfrey Nolan on-line!
See this thread for details.
The moose likes Java in General and the fly likes java heap size out of memory error Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Java » Java in General
Bookmark "java heap size out of memory error" Watch "java heap size out of memory error" New topic
Author

java heap size out of memory error

manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
Hi All,

I was wondering if there is a better way to handle a large for loop's processing. Currently I have a for loop which processes 50k records. The way we get the 50k candidate records to process is before the for loop via database query. This query is a time consuming process and sometimes if the records pulled exceed 50k we run into the following error
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.

What is the best way to deal with this should we request more memory on the app server or should we resort to improvisations in the code.
How to determine what heap size to specify in the app server configurations.

If we need to counter this in the code what would be the best way to do it. Is there anyway we can do the query pull in a staggered fashion, get the records in series of chunks one after the other and pass it on to the for loop process.

Please advise

Thanks,
Stephan van Hulst
Bartender

Joined: Sep 20, 2010
Posts: 3599
    
  14

What kind of records are they and what do you plan on doing with them?
manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
These records are a collection of Hibernate POJO objects. We only fetch the column values which are absolutely necessary for processing.
Stephan van Hulst
Bartender

Joined: Sep 20, 2010
Posts: 3599
    
  14

You need to be more specific. What is the purpose of the records, or in this case, these columns? What processing do you need to do?
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18541
    
    8

And why do you need to have them all in memory at the same time?
manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
sorry for the partial info.

Stephan:
The database query to pull the records is kind of a feeder service functionality. Inside the for loop these records are further processed for all sort of java business logic calculations and the output stored in a database table for each such record retrieved.

Paul:
yes thats the question I meant to ask initially. Can I stagger the records retrieved that i fetch upfront currently. I don't need them all together but since earlier the volumes were low and we could retrieve them all at once we fetched everything in one go.

Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18541
    
    8

manish ahuja wrote:Paul:
yes thats the question I meant to ask initially. Can I stagger the records retrieved that i fetch upfront currently. I don't need them all together but since earlier the volumes were low and we could retrieve them all at once we fetched everything in one go.


I would say you should process the records one at a time, rather than loading them all into memory first. That's provided your application logic allows that, of course. Whether you can depends on a lot of things, none of which we can see from here.
manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
We could but the additional database round-trip to fetch each record in this case will have a significant impact on the performance.
I was thinking on the lines of pagination where initially we only get the total record count to process and then divide them by say 10 to get discreet chunk sizes like first 10000, 10001-20000, 20001-30000 so on to process.

The records represented by individual chunks will then be pulled and passed on to the for loop and once this for loop ends we fetch the next set of records to process.
My only concern is how we can maintain the point to get the next set of records.


fred rosenberger
lowercase baba
Bartender

Joined: Oct 02, 2003
Posts: 11161
    
  16

I may be totally wrong here, as this is not my area. But this sounds like a situation where you'd use a cursor (or at least you'd use what I believe a cursor to be).

A cursor is like a pointer into a query. it lets the database get the results and then you use the cursor to iterate through them. I don't know what the performance hit would be, but i imagine you could possibly set it up, have one thread getting the records into a buffer, and then have other threads processing them off. your getter would make sure there is always work for the processors to do by getting the next record each time the count in the buffer goes below a certain threshold (assuming there are any left to get).




There are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18541
    
    8

manish ahuja wrote:We could but the additional database round-trip to fetch each record in this case will have a significant impact on the performance.

What's this extra round-trip you're talking about?

Normally you would do something like this:


There's aren't any "round-trips" in that code.

But you say you're using Hibernate? Probably Hibernate provides some advanced and convoluted way to do that simple thing, but I'm no Hibernate expert. If it doesn't, you might want to ask whether Hibernate is providing a net benefit to your design.
manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
I think I didn't explain the context clearly. To stay with your example


ResultSet rs = //number of results from the query less than 50K works fine
while(rs.next())
{
// processing goes here
}



ResultSet rs = // number of results from the query is greater than 50K and fails with java heap out of memory error

while(rs.next())
{
// processing goes here
}

Now what i want to do is split the rs database query records into chunk sizes if greater than 50k and pull the records in smaller volumes from the database to work around the heap memory error.

paginate the resultset (retrieve a chunk of records) and for each pagination execute the for loop for the records retrieved.




Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18541
    
    8

Then you are building the ResultSet in such a way that all of its records get loaded into memory at the same time. Try to avoid doing that.
Mohamed Sanaulla
Saloon Keeper

Joined: Sep 08, 2007
Posts: 3068
    
  33

Pagination would be good option. You can store the number of records retrieved each time. That way each time you can select the number of rows such that the count is <= the number of records retrieved each time.

Agree there would be performance issues of hitting the dB multiple times for obtaining the ResulSet, but atleast better than coming across OutOfMemoryException.


Mohamed Sanaulla | My Blog
Ravi Kiran Va
Ranch Hand

Joined: Apr 18, 2009
Posts: 2234

Hi Manish ,

If you are using Oracle as your Database , try googling with rownum .


Save India From Corruption - Anna Hazare.
himanshu.harish agrawal
Ranch Hand

Joined: Oct 18, 2010
Posts: 47

Paul Clapham wrote:Then you are building the ResultSet in such a way that all of its records get loaded into memory at the same time. Try to avoid doing that.


Sir, can you please elaborate your point.

What i think is that all the information will be retrieved and get stored in the ResultSet object. So, is it that the size of this object becomes very large, after getting more than 50K records, and hence heap is going outofmemory?
And if so then will running GC little earlier then this will help in solving the problem.

Please reply.

Thanks.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
 
subject: java heap size out of memory error
 
Similar Threads
JSPCompilation Error
Java code File and FileWriter
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: Java heap space
performance