This week's book giveaway is in the Mac OS forum.
We're giving away four copies of a choice of "Take Control of Upgrading to Yosemite" or "Take Control of Automating Your Mac" and have Joe Kissell on-line!
See this thread for details.
The moose likes JDBC and the fly likes Huge database access Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login

JavaRanch » Java Forums » Databases » JDBC
Bookmark "Huge database access" Watch "Huge database access" New topic

Huge database access

Arjun Shastry
Ranch Hand

Joined: Mar 13, 2003
Posts: 1874
We are developing an integration software which basically integrates between one software(which collects alot of data(from various computer nodes in a network) and puts/updates in a database.Our part is to retrieve this data and put it into another Assetmaangement software through API.There is no problem when dataset is small i.e. say 25000 total records.What algorithms or techniques are used while accessing tables containing millions of records?If we use the same,then it will be a probelm with JVM heap size,which occurs many times.We can increase the heap size,but want to know any other approach?

SJ Adnams
Ranch Hand

Joined: Sep 28, 2001
Posts: 925
outsource to india?
Joe Ess

Joined: Oct 29, 2001
Posts: 8915

And your Indian oursourcers will probably use a paging technique to get a block of records, process it, then retrieve the next block.

"blabbing like a narcissistic fool with a superiority complex" ~ N.A.
[How To Ask Questions On JavaRanch]
Arjun Shastry
Ranch Hand

Joined: Mar 13, 2003
Posts: 1874
Well,we don't depend on people and client who are 5000 miles away and conducting video conferences.[This will be a part of project for Indian Railway]
[ December 09, 2003: Message edited by: Capablanca Kepler ]
GeeCON Prague 2014
subject: Huge database access