aspose file tools*
The moose likes Performance and the fly likes Arraylist vs heapsize: Out of memory error Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Performance
Bookmark "Arraylist vs heapsize: Out of memory error" Watch "Arraylist vs heapsize: Out of memory error" New topic
Author

Arraylist vs heapsize: Out of memory error

Kishore Polsani
Greenhorn

Joined: May 20, 2012
Posts: 11
Hi All,

I have an arraylist which will hold nearly 30,000 elements. When I ran the java program it is throwing "java/lang/OutOfMemoryError". I have changed heap size to 1GB then the job worked fine after sometime again it throwed the same error. Please help me how can I get rid of this.

Command: java -Xmx1024m sample

Error:
JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait.

Thanks,
Kishore
fred rosenberger
lowercase baba
Bartender

Joined: Oct 02, 2003
Posts: 11476
    
  16

Don't hold so many objects in memory.


There are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors
Tim Cooke
Bartender

Joined: Mar 28, 2008
Posts: 1173
    
  65

and if you really really must hold 30,000 objects in memory (unlikely), then make them really really small objects.


Tim Driven Development
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12823
    
    5
Note that the automatic enlargement for ArrayList roughly doubles the size of the underlying array and does a copy.

The default initial size is 10 - just for fun calculate the number of double/enlarge cycles to get to 30,000!

Thats right, grossly inefficient and makes lots of memory chewing objects.

Instead create your initial ArrayList slightly larger than you will ever need - much faster and easier on memory.

Bill
Ganesh Malaichamy
Greenhorn

Joined: Feb 11, 2011
Posts: 27
Also revisit the code design for your use case. Do you really need to hold 30000 objects for the processing at the same time.You can sub-task the processing based on your actual use case.serializing is one way to do it.
Martin Vajsar
Sheriff

Joined: Aug 22, 2010
Posts: 3611
    
  60

William Brogden wrote:The default initial size is 10 - just for fun calculate the number of double/enlarge cycles to get to 30,000!

Thats right, grossly inefficient and makes lots of memory chewing objects.

I'm not sure it is that clear-cut.

It would take some 12 reallocations to grow from 10 to 30,000 capacity according to the parameters you've specified. But the ArrayList actually grows by 1.5, not 2 every time, so it will be some 20 reallocations, and it will allocate three times the resulting capacity in the process. Two third of the memory will be freed, of course, and eventually garbage collected. It will amount to some 300 KB of memory in our case (assuming 32bit JVM, in 64 bit JVM it could be two times as much, unless the JVM performed some tricks to decrease the size of a pointer).

In my experience the growing process is pretty fast. Memory allocation is really, really fast in Java (took me a long time to start believing it ). Copying the old array to the new is also pretty fast - it's even done by a native method.

So, the benefits will not really be measurable in a profiler in most applications. But, there can be downsides!

If the needed capacity of the list is only being guessed at and somehow hardcoded into the code, it may easily happen that the guess will become incorrect as the code evolves. If the guess is smaller than the actual need, no harm is done and even some time/memory (update: it's more complicated with memory) is still saved (compared to using the default capacity). But I've experienced changes in code that made my guesses absurdly exaggerated. And this is where the penalty is - if I've expected a list to contain 10,000 elements when I've coded it, but then reorganized the data into 200 lists of 50 elements each and forgot to change the initial capacity guess, I'm now wasting some 1,900,000 object references (about 8 MB in 32 bit JVM)!. And unlike the first case, this memory is allocated for the lifetime of the entire ArrayList (or until trimToSize() is called), whereas in the first case most of the memory allocated in excess is immediately freed.

I've discovered this problem in my code some time ago, and I've found I'm not reliably able to review and update all relevant capacity guesses after making changes to the application, I've stopped specifying the capacity in most cases. I still specify the capacity when I'm creating a collection by processing another existing collection whose size can be queries and I expect to transfer most items over. I would also specify the capacity if I was expecting the list to be so large that it would occupy significant portion of the memory (so staring at least at a million), but I didn't have such a need in production code yet. Nowadays I believe that striving to always specify the capacity is actually an example of premature optimization.

Instead create your initial ArrayList slightly larger than you will ever need - much faster and easier on memory.

This is the key point. If I could always make sure that the specified capacity will be at most slightly larger, everything would be neat and dandy. But I found it surprisingly hard and time-consuming making sure it is so.

Just my two cents
Martin Vajsar
Sheriff

Joined: Aug 22, 2010
Posts: 3611
    
  60

Back to the original question: if the code worked fine for some time and then crashed on the memory, it sounds more like a memory leak to me.

Also, I'd say 30,000 objects in total isn't that much. But 30,000 objects per user in a web application (for example) is certainly way too much.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Arraylist vs heapsize: Out of memory error