wood burning stoves
The moose likes EJB and other Java EE Technologies and the fly likes Retrieval of Huge amount of Data Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » EJB and other Java EE Technologies
Bookmark "Retrieval of Huge amount of Data" Watch "Retrieval of Huge amount of Data" New topic

Retrieval of Huge amount of Data

thomas davis
Ranch Hand

Joined: Feb 01, 2003
Posts: 207
I need to retrieve a huge amout of data to my application.How do I accomplish this task without any performance trade-offs and bottlenecks?

Do I need to use Cursor or Stored Procedure OR Can I do something from Front End?

Do I need to pull it in chunks?If so how do I do set place holder or set the handler?
jeetu krishnan

Joined: Feb 13, 2006
Posts: 12
We faced multiple issues when trying to display large amount of data for a screen.
Some of them include..
1) poor performance on the database side.
2) Screen rendering taking a lot of time.
3) If there is business logic to be executed on the data retrieved from the database you might face memory issues....etc

I am really not sure how it will be beneficial to display large amount of data on a single screen to the end user for your use case.

The better option is to implement some pagination logic which will show maller chunks of data and thereby eliminate any screen rendering ormemory related issues.

Another option is to use Ajax so that the client perception also improves.
Romario Dominic

Joined: Feb 27, 2006
Posts: 26
It should definitely help to have a StoredProcedure at the back end.A lot can be done at the back end to improve performance for eg:Having indexes.

Once you get the data on the front end,the display could be handled however you wanted.

Not sure if JMS could be used to pull the data out.
Devender Thareja
Ranch Hand

Joined: Jul 14, 2005
Posts: 187
In addition to above..
I would guess you don't want to display huge amount of data, you need to process it instead. If that's the case,
just use jdbc preparedStatement and avoid creating objects for each loop (I think it's called flyweight pattern) and do the processing.
Use scrollable ResultSet if there is a need for that.
You can use more then one connection and process them in parrellel.
Consider prefetching data.
Hope it helps.

Devender Thareja
I agree. Here's the link: http://aspose.com/file-tools
subject: Retrieval of Huge amount of Data
It's not a secret anymore!