File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
The moose likes Object Relational Mapping and the fly likes large collections in hibernate. Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Databases » Object Relational Mapping
Bookmark "large collections in hibernate." Watch "large collections in hibernate." New topic

large collections in hibernate.

jay vas
Ranch Hand

Joined: Aug 30, 2005
Posts: 407
Hi guys. I am trying to do a query on a table with four million records.
I've set everything to "lazy" and additionally have even tried
setting cachemode to IGNORE.

Neither effort seems to be saving me. I'm still getting the following dreaded error : Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.

This seems like it should be simple. Is there a configuration setting im missing ?

Shailesh Kini
Ranch Hand

Joined: Oct 17, 2001
Posts: 153
Hey Jay,

It is not clear if the table protein has 4 mil rows or the related tables have 4 million rows?

You should add filtering to your query. If you are interesting in a subset of the records use JPQL to build your entities.

You could also try to increase the max heap space on your jvm.

Shailesh Kini.
jay vas
Ranch Hand

Joined: Aug 30, 2005
Posts: 407
The table which i am querying directly has 4 million rows.
I don't want to fix the max heap size, because I assume one day the table may have 10 million rows, etc. I would rather have a scalable solution which is lazy. Does hibernate not provide a lazy iterator for a result ?

I need to iterate through all "n" records, regardless of how large n is, so I cannot want to filter the query in any way.

Im shocked if this is the case !

Please let me know.
Elanges waran

Joined: Dec 13, 2007
Posts: 17
Hi Jay Vas,

My suggestions are avoid this scenario as follows:

* use Hibernate pagination concepts with page by page records showing.

* Also Introduce DB VIEW and module for getting records using HQL.
[Check for this : ]]

Give your feedback...

Regards,<br />K.Elanges
Rama Krishna
Ranch Hand

Joined: Oct 16, 2007
Posts: 110
The link seems to be not working
John Grath

Joined: Sep 18, 2007
Posts: 15

You should limit the size of what is being returned from the database. Try appending a "where clause" to the SQL and see if that works. You should not be reading 4m records!!


Mark Spritzler

Joined: Feb 05, 2001
Posts: 17276

If you are going to be iterating through tons of records, I highly recommend that you evict objects as you are finished reading them. This will clear stuff out of the Persistence Context maps and not store all the record in memory.

So since you know that the collection loads as you go through each record, because of lazy fetching and the fetchmode isn't eager or subselect.

So as you are looping through them and going to the database, after you are done with that record, then evict the object from the session.


Perfect World Programming, LLC - iOS Apps
How to Ask Questions the Smart Way FAQ
I agree. Here's the link:
subject: large collections in hibernate.
It's not a secret anymore!