aspose file tools*
The moose likes Performance and the fly likes Huge Collection: how to prevent OutOfMemoryError? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Soft Skills this week in the Jobs Discussion forum!
JavaRanch » Java Forums » Java » Performance
Bookmark "Huge Collection: how to prevent OutOfMemoryError?" Watch "Huge Collection: how to prevent OutOfMemoryError?" New topic
Author

Huge Collection: how to prevent OutOfMemoryError?

Mike Nason
Greenhorn

Joined: Apr 19, 2006
Posts: 20
Hello!
I just interested in one thing: is it possible to put a huge number of elements in collection without having OutOfMemoryError.

I am writting a small swing program and need generate random keys=>values dictionary. I use hashtable(which is syncronized) and thread pool to generate and put a given number of elements.
If I try to generate some thousand of elements and put it into collection it works fine.
If the number will be too big (some millions) then the program starting slow down, the swing UI is not responsible and it crashes.

Are there any ways to make at least swing GUI responsible and store a big number of elements into collection?

Thanks.
David O'Meara
Rancher

Joined: Mar 06, 2001
Posts: 13459

The read time of a java.util.HashTable shouldn't degrade as it grows, but part of that depends on the implementation of the key class.
The numbers here might help.
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12835
    
    5
You can make creating a collection faster by constructing it with a large enough starting size. Java collections that automatically grow do so by creating new arrays and copying the old references. This is not efficient for "huge" collections and at one point you have to have both the old and new data structure in memory, causing OutOfMemoryError unnecessarily.

Bill
Mike Nason
Greenhorn

Joined: Apr 19, 2006
Posts: 20
What happens if I have a Key object like shown below and doesn't have equals() and hashCode() methods overriden there.
Does it reduce performance?

My Key object:


Another words I have a Key as a combination of long value and 5Character String.
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18987
    
    8

Alex Shvedov wrote:What happens if I have a Key object like shown below and doesn't have equals() and hashCode() methods overriden there.
Does it reduce performance?


You mean reduce performance compared to implementing equals() and hashCode(), I suppose. Why don't you try it and see?
Pat Farrell
Rancher

Joined: Aug 11, 2007
Posts: 4659
    
    5

Alex Shvedov wrote:I just interested in one thing: is it possible to put a huge number of elements in collection without having OutOfMemoryError.

Its easy, just have a humongous amount of real memory.

You really have two separate problems here.
1) your performance goes down with lots of entries
2) you run out of memory with lots of entries

The solution to #2 is either give the JVM tons of memory, or don't store the objects in memory, that is why there are disks.

The solution to #1 is to not use a random access based on the key. First, use something besides a random number as the key, so you can cluster values. You would be better off using a structure such as a red/black tree.

And implement equals() and hashCode() if you stay with any of the hash* structures. The default JVM implementation uses the address of the object in side the JVM.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Huge Collection: how to prevent OutOfMemoryError?