File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
The moose likes JDBC and Relational Databases and the fly likes Huge database access Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Databases » JDBC and Relational Databases
Bookmark "Huge database access" Watch "Huge database access" New topic

Huge database access

Arjun Shastry
Ranch Hand

Joined: Mar 13, 2003
Posts: 1893
We are developing an integration software which basically integrates between one software(which collects alot of data(from various computer nodes in a network) and puts/updates in a database.Our part is to retrieve this data and put it into another Assetmaangement software through API.There is no problem when dataset is small i.e. say 25000 total records.What algorithms or techniques are used while accessing tables containing millions of records?If we use the same,then it will be a probelm with JVM heap size,which occurs many times.We can increase the heap size,but want to know any other approach?

SJ Adnams
Ranch Hand

Joined: Sep 28, 2001
Posts: 925
outsource to india?
Joe Ess

Joined: Oct 29, 2001
Posts: 9168

And your Indian oursourcers will probably use a paging technique to get a block of records, process it, then retrieve the next block.

[How To Ask Questions On JavaRanch]
Arjun Shastry
Ranch Hand

Joined: Mar 13, 2003
Posts: 1893
Well,we don't depend on people and client who are 5000 miles away and conducting video conferences.[This will be a part of project for Indian Railway]
[ December 09, 2003: Message edited by: Capablanca Kepler ]
I agree. Here's the link:
subject: Huge database access
It's not a secret anymore!