This week's book giveaway is in the Cloud/Virtualizaton forum.
We're giving away four copies of Mesos in Action and have Roger Ignazio on-line!
See this thread for details.
Win a copy of Mesos in Action this week in the Cloud/Virtualizaton forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Huge Collection: how to prevent OutOfMemoryError?

 
Mike Nason
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello!
I just interested in one thing: is it possible to put a huge number of elements in collection without having OutOfMemoryError.

I am writting a small swing program and need generate random keys=>values dictionary. I use hashtable(which is syncronized) and thread pool to generate and put a given number of elements.
If I try to generate some thousand of elements and put it into collection it works fine.
If the number will be too big (some millions) then the program starting slow down, the swing UI is not responsible and it crashes.

Are there any ways to make at least swing GUI responsible and store a big number of elements into collection?

Thanks.
 
David O'Meara
Rancher
Posts: 13459
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The read time of a java.util.HashTable shouldn't degrade as it grows, but part of that depends on the implementation of the key class.
The numbers here might help.
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13061
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You can make creating a collection faster by constructing it with a large enough starting size. Java collections that automatically grow do so by creating new arrays and copying the old references. This is not efficient for "huge" collections and at one point you have to have both the old and new data structure in memory, causing OutOfMemoryError unnecessarily.

Bill
 
Mike Nason
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What happens if I have a Key object like shown below and doesn't have equals() and hashCode() methods overriden there.
Does it reduce performance?

My Key object:


Another words I have a Key as a combination of long value and 5Character String.
 
Paul Clapham
Sheriff
Posts: 21107
32
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Alex Shvedov wrote:What happens if I have a Key object like shown below and doesn't have equals() and hashCode() methods overriden there.
Does it reduce performance?


You mean reduce performance compared to implementing equals() and hashCode(), I suppose. Why don't you try it and see?
 
Pat Farrell
Rancher
Posts: 4678
7
Linux Mac OS X VI Editor
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Alex Shvedov wrote:I just interested in one thing: is it possible to put a huge number of elements in collection without having OutOfMemoryError.

Its easy, just have a humongous amount of real memory.

You really have two separate problems here.
1) your performance goes down with lots of entries
2) you run out of memory with lots of entries

The solution to #2 is either give the JVM tons of memory, or don't store the objects in memory, that is why there are disks.

The solution to #1 is to not use a random access based on the key. First, use something besides a random number as the key, so you can cluster values. You would be better off using a structure such as a red/black tree.

And implement equals() and hashCode() if you stay with any of the hash* structures. The default JVM implementation uses the address of the object in side the JVM.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic