Lots of Data :: Out of Memory

[Bob Mathews]

Hi all Ranchers,

This is my first post in this forum. I have a big performance issue in my System. Here is the description:-
Based ion some parameters call is send to the server and then to DB, All the data is fetched in the resultset and is given to the customized Iterator Object. This data is then given to a component which can render data in the tabular format, just like the way it is in the main page of this forum.

Id Name Description Other_Information --- --- ---

and down at the bottom there is a scope of pagination just like this

1 2 3 4 5 6 7 8 9 10 next previous First Last

Now the issue is data set can go upto a million records and we are getting the whole lot of data from the DB in one go.

Secondly the framework to render data in the Tabular format, renders each cell as an object either a custom Object or String or Date etc

And most of the time we run into Java Out of Memory exception

Can you people please suggest something on this.

Is there any solution related to Caching the data

How to implement pagination in Performance effective way

Is there anything that can be done about the rendering framework something like to render data in some basic data types and not in Custom objects.


Please suggest on this ASAP

Thanks in Advance
Nishant

[Rejeev Divakaran]

Now the issue is data set can go upto a million records and we are getting the whole lot of data from the DB in one go

You can try query like below to introduce paging in DB layer.
select * from emp where empid < ?1 and rownum < 100 order by empid
You need to pass the empid of last returned row to this query.

[Peter Chase]

Originally posted by Nishant Tiwari:
This is my first post in this forum.

"Philanthropic",

Welcome to JavaRanch.

When you registered, you may not have seen that JavaRanch has a naming policy, which is mandatory and will be enforced, if necessary. You must use a real name (or at least one that sounds real) as your Display Name.

Please go to the "My Profile" link and change your Display Name to a real (sounding) name.

[mohan shetty]

This type of problem with regard to performance i had faced once.. if it contains lots of data to be fetched, see those data is coming from a single or from more tables with some comparision.here noramalisation with many tables may cause performance issues so my suggetion is try to remove the normalisation if possilble or if its a case of comparision with some other fields of other tables,the data which you are retrieving..there try to remove OR clause and use WHERE clause..this may work..

[Pat Farrell]

This is fundamentally a "doctor doctor" problem.
The punch line is "don't do that"

So then we have to ask, what are you doing that is so slow. Without more details, I can only guess, but its usually one of two things:

One: if you are doing a nine-way join, as mohan shetty suggested, don't do that.

Two: If you are just doing a select from a single table, then it is more complicated, but realistically, no one ever wants to look at a million rows returned, so don't do that either. Change your code to pull in blocks of rows, just a few times more than the user will ever want. So if 50 fit on a page, get 100 and keep track. If only 5 fit, get 10 to 20