Win a copy of Think Java: How to Think Like a Computer Scientist this week in the Java in General forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Java Heap Space Error, OutofMemory Exception while writing large data to excel sheet.

 
Prashant Kadake
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
I am getting Java Heap Space Error while writing large data from database to an excel sheet.
I dont want to use JVM -XMX options to increase memory.

Following are the details:
1) I am using org.apache.poi.hssf api for excel sheet writing.
2) JDK version 1.5
3) Tomcat 6.0
Code i have wriiten works well for around 23 thousand records, but it fails for more than 23K records.

Following is the code:



Please help me with the solution.
 
Paul Sturrock
Bartender
Posts: 10336
Eclipse IDE Hibernate Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

I dont want to use JVM -XMX options to increase memory.

Why not? It sounds like you probably need to increase the heap.
 
Ulf Dittmer
Rancher
Posts: 42967
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yep, POI uses a lot of memory when dealing with large spreadsheets. Having to up the memory is a common occurrence.
 
Prashant Kadake
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
@Paul Sturrock I think i should work on code, optimized enough to cope with default memory of JVM. Having good hope that some engineering can save me.! can you please go through the code and suggest me some good practises to handle memory usages.
@Ulf Dittmer May be POI can be the reason for outOfMemory, but can you please go through the code and suggest me some good practises to handle memory usages.
 
Sai Hegde
security forum advocate
Ranch Hand
Posts: 212
1
Android Flex Google App Engine
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I havent run through your code but good design patterns most of the times should keep you from falling into such exceptions.
Did you run your code against a fairly smaller amount of data in your excel sheet. Use the Flyweight Pattern to create your objects if possible.
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13061
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How about writing the data as CSV lines and stream them to the user? Very low memory use and much faster than POI object creation. Excel should be happy to read CSV and build the spreadsheet.

There appears to be no real reason to build POI objects.

Bill
 
Paul Clapham
Sheriff
Posts: 21107
32
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Prashant Kadake wrote:I think i should work on code, optimized enough to cope with default memory of JVM.


Are you're suggesting that it should be possible to modify any code whatsoever to run in any limited amount of memory? That may be true in theory but it may require a lot of work in real life. For example POI is written to keep its data in memory, so the more data you give it, the more memory it uses. It might be possible to modify POI to use temporary disk files (for example) as a backing store, but that would probably take an experienced programmer several months to do.
 
Sumit Bisht
Ranch Hand
Posts: 331
Python Ruby Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Prashant, I am not familiar with POI, but it looks as if you are using the entire dataset in a single chunk. Can you try doing that in different steps ?
Also, have a look at Weakly referenced (http://download.oracle.com/javase/1.4.2/docs/api/java/lang/ref/WeakReference.html) datastructures in this case.
 
Jimmy Clark
Ranch Hand
Posts: 2187
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't see any mention of it in this thread, but have you analyzed the JRE's memory usage in detail. The details of this analysis would help identify where and if you have memory leaks in your code and where you can possibly rewrite the code to be more efficient.
 
David O'Meara
Rancher
Posts: 13459
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Eclipse has a Memory Analysis Tool which explains how to get a memory dump and how to use this to track where all of the memory is taken up.
http://www.eclipse.org/mat/

Saying that you 'don't want to increase memory' may not be an option. Sometimes if the application needs more memory, it needs more memory.
 
Jimmy Clark
Ranch Hand
Posts: 2187
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
NetBeans has a high-end powerful Java Profiler suited for analyzing a JRE. Check out the details:

http://netbeans.org/features/java/profiler.html
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic