File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

How to get rid of heap OutofMemoryError

 
Vinny Menon
Ranch Hand
Posts: 62
Java Oracle Spring
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi
I am posting after a long time..Hope everyone is fine
I am using Eclipse as my IDE.
In my application,i am creating an arraylist of my custom object(e.g CustomTransactionObject).The object has around 20 member fields.

I get around 57896 records from the database.each record represents one CustomTransactionObject.
i have to map those values to the CustomTransactionObject.



in this code..when the record count reaches around 22525 ..it gives me an OutofMemoryError.
I tried to change the settings of Eclipse by changing the vmargs parameter from 256M to 512M (ref eclipse.ini)

but still i got the same error..

can you please suggest me a solution or should i go for any other approach

cheers
vinnym


"feels good to be back after a long absence"
 
Mike Noel
Ranch Hand
Posts: 108
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
When you switched to 512 did it still crash at about 22000 records? Also, how big are the fields in the class. In order to to run out of memory due to these objects it seems that each one would need to be around 10k in size. Is that about what you're dealing with?

If your records really are that big then you might want to think of a different algorithm for processing them. Even if you solve the memory problem for now your solution won't scale to more data.
 
Rusty Shackleford
Ranch Hand
Posts: 490
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You might want to think about an external ADT , like a B-tree
 
Maulin Vasavada
Ranch Hand
Posts: 1873
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Rusty

This is little off the thread here but can you really give me example of real life use of B-tree in java code...? Somehow I never could hold of such thing. I know why one would want to use B-tree (for the same reason Vinny is facing OOM Error here) but never could fit to my applications so far

Thanks
Maulin
 
Naseem Khan
Ranch Hand
Posts: 809
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Originally posted by Vinny Menon:
In my application,i am creating an arraylist of my custom object(e.g CustomTransactionObject).The object has around 20 member fields.

I get around 57896 records from the database.each record represents one CustomTransactionObject.
i have to map those values to the CustomTransactionObject.



Why you are retrieving that much record from database. Will you show 57,896 records to the end user at one time. obviously not.

You are getting OutOfMemoryError when only one thread is calling databse. What will happen when suppose 100 threads simultaneously call your dao component. You will have 57 lacs 89 thousand 600 instances in single jvm.

Better approach is retrieve only the required data. Say in first page if you want to show 10 records, retrieve 10 rows only. When user want to see more record say from 11-20, then make another call to database and retrieve 11-20 rows.

This approach is called pagination.

More on this check this thread...

Pagination


Naseem
 
Vinny Menon
Ranch Hand
Posts: 62
Java Oracle Spring
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi
thanks for your replies..
Why you are retrieving that much record from database. Will you show 57,896 records to the end user at one time. obviously not.

You are getting OutOfMemoryError when only one thread is calling databse. What will happen when suppose 100 threads simultaneously call your dao component. You will have 57 lacs 89 thousand 600 instances in single jvm.

Better approach is retrieve only the required data. Say in first page if you want to show 10 records, retrieve 10 rows only. When user want to see more record say from 11-20, then make another call to database and retrieve 11-20 rows.


naseem, i do not have to show these records on the front end to the user,if i had to show the records, then probably pagination is the best approach,as you have suggested.
I have implemented pagination in my earlier projects,which involved front end .

Instead i have to generate a report(essentially write to a file and provide the file to the user)..
Also i am using a single thread to retrieve data.Also at anytime only one thread will be active for this.

I am still pondering on how to solve this problem :roll:

Thanks
vinny m
 
Vj Kamath
Greenhorn
Posts: 15
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Vinny wrote:
i do not have to show these records on the front end to the user,if i had to show the records, then probably pagination is the best approach,as you have suggested.
I have implemented pagination in my earlier projects,which involved front end .

Instead i have to generate a report(essentially write to a file and provide the file to the user)..
Also i am using a single thread to retrieve data.Also at anytime only one thread will be active for this.


You could use pagination for storing in the file as well. The concept is the same. You just retrieve a fixed set of records and write to the file(or report) and clearup the objects, then take the next set.

Advantage: You will used fixed amount of memory. Memory will not be dependant on the number of records you are retrieving

Disadvantage: You will have to make multiple calls to the database instead of one call. This may impact the response time for generating that report.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic