Hi I am posting after a long time..Hope everyone is fine I am using Eclipse as my IDE. In my application,i am creating an arraylist of my custom object(e.g CustomTransactionObject).The object has around 20 member fields.
I get around 57896 records from the database.each record represents one CustomTransactionObject. i have to map those values to the CustomTransactionObject.
in this code..when the record count reaches around 22525 ..it gives me an OutofMemoryError. I tried to change the settings of Eclipse by changing the vmargs parameter from 256M to 512M (ref eclipse.ini)
but still i got the same error..
can you please suggest me a solution or should i go for any other approach
cheers vinnym
"feels good to be back after a long absence"
Regards,Vinny M.
proud Fan of European Champion CHELSEA FC
"If you don't see the bug where you're looking, perhaps you're looking in the wrong place" -James Gosling
Mike Noel
Ranch Hand
Joined: Dec 15, 2005
Posts: 108
posted
0
When you switched to 512 did it still crash at about 22000 records? Also, how big are the fields in the class. In order to to run out of memory due to these objects it seems that each one would need to be around 10k in size. Is that about what you're dealing with?
If your records really are that big then you might want to think of a different algorithm for processing them. Even if you solve the memory problem for now your solution won't scale to more data.
You might want to think about an external ADT , like a B-tree
"Computer science is no more about computers than astronomy is about telescopes" - Edsger Dijkstra
Maulin Vasavada
Ranch Hand
Joined: Nov 04, 2001
Posts: 1865
posted
0
Hi Rusty
This is little off the thread here but can you really give me example of real life use of B-tree in java code...? Somehow I never could hold of such thing. I know why one would want to use B-tree (for the same reason Vinny is facing OOM Error here) but never could fit to my applications so far
Originally posted by Vinny Menon: In my application,i am creating an arraylist of my custom object(e.g CustomTransactionObject).The object has around 20 member fields.
I get around 57896 records from the database.each record represents one CustomTransactionObject. i have to map those values to the CustomTransactionObject.
Why you are retrieving that much record from database. Will you show 57,896 records to the end user at one time. obviously not.
You are getting OutOfMemoryError when only one thread is calling databse. What will happen when suppose 100 threads simultaneously call your dao component. You will have 57 lacs 89 thousand 600 instances in single jvm.
Better approach is retrieve only the required data. Say in first page if you want to show 10 records, retrieve 10 rows only. When user want to see more record say from 11-20, then make another call to database and retrieve 11-20 rows.
Why you are retrieving that much record from database. Will you show 57,896 records to the end user at one time. obviously not.
You are getting OutOfMemoryError when only one thread is calling databse. What will happen when suppose 100 threads simultaneously call your dao component. You will have 57 lacs 89 thousand 600 instances in single jvm.
Better approach is retrieve only the required data. Say in first page if you want to show 10 records, retrieve 10 rows only. When user want to see more record say from 11-20, then make another call to database and retrieve 11-20 rows.
naseem, i do not have to show these records on the front end to the user,if i had to show the records, then probably pagination is the best approach,as you have suggested. I have implemented pagination in my earlier projects,which involved front end .
Instead i have to generate a report(essentially write to a file and provide the file to the user).. Also i am using a single thread to retrieve data.Also at anytime only one thread will be active for this.
I am still pondering on how to solve this problem :roll:
Thanks vinny m
Vj Kamath
Greenhorn
Joined: Nov 19, 2004
Posts: 15
posted
0
Vinny wrote:
i do not have to show these records on the front end to the user,if i had to show the records, then probably pagination is the best approach,as you have suggested. I have implemented pagination in my earlier projects,which involved front end .
Instead i have to generate a report(essentially write to a file and provide the file to the user).. Also i am using a single thread to retrieve data.Also at anytime only one thread will be active for this.
You could use pagination for storing in the file as well. The concept is the same. You just retrieve a fixed set of records and write to the file(or report) and clearup the objects, then take the next set.
Advantage: You will used fixed amount of memory. Memory will not be dependant on the number of records you are retrieving
Disadvantage: You will have to make multiple calls to the database instead of one call. This may impact the response time for generating that report.
0
