This week's book giveaway is in the JavaScript forum.
We're giving away four copies of JavaScript Promises Essentials and have Rami Sarieddine on-line!
See this thread for details.
The moose likes JDBC and the fly likes How I can retrive 40 million records from Database? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of JavaScript Promises Essentials this week in the JavaScript forum!
JavaRanch » Java Forums » Databases » JDBC
Bookmark "How I can retrive 40 million records from Database?" Watch "How I can retrive 40 million records from Database?" New topic
Author

How I can retrive 40 million records from Database?

Sami Ahmad
Greenhorn

Joined: May 18, 2010
Posts: 9
Hello all,

please I am trying to retrieve 40 million records from Oracle Database, but my JSF page still loading without retrieve anything.

FYI, I use SQL statement to retrieve these data.

So is there any tools help. (hadoop Apache software not supported for Windows Production, only for development)

Thank you in advance.

Sami.


Best Regards.

SCJP 6 SCWCD 5
Tom Reilly
Rancher

Joined: Jun 01, 2010
Posts: 618
Because you are using JSF, I assume you want to display the information to a user on a web page. I'll skip asking why a user would want to see 40 million records. Does your app work when you try to display one record? If you post your code, maybe someone can comment on it.
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 42584
    
  65
Tom Reilly wrote:I'll skip asking why a user would want to see 40 million records.

I won't :-)

As Tom said, so many records should not be handled by the web layer because there's no way for a user to view them. They most likely should not be handled in one batch no matter what: if a record takes just 100 bytes of space then ... well, you do the math :-)


Ping & DNS - my free Android networking tools app
Sami Ahmad
Greenhorn

Joined: May 18, 2010
Posts: 9
Thank you all, every day my customer sending 300000 messages to his customers, So he want to know what he send before 3 month.
let's say the point not see 40 million records, bcz I can make pagging in my JSF page that will retrieve 500 every search.

but when I am trying to make search with some conditions like From (Date), To (Date), Status, etc.... the page still loading without retrieve anything.

FYI.. the page retrieve 2 million records in 1 min .

Thank you again.
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 42584
    
  65
Does the search query work if you run it directly against the DB (using the DB's tools)?
Sami Ahmad
Greenhorn

Joined: May 18, 2010
Posts: 9
Thank you,
I am using Toad, it retrieve 500 row when I make search, I want to know if there tools or software store huge number of data dealing with java language, but NOT hadoop.

Sami .
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 42584
    
  65
I think you should rethink your design. There is no point for the search query to return more records than will be displayed, and IMO 500 records is too many to put on a page meant for human consumption; consider displaying 50 records, or 100 at the most.
Scott Selikoff
author
Saloon Keeper

Joined: Oct 23, 2005
Posts: 3716
    
    5

I think for fun you should create a dummy page that just create 40 million rows, repeated without going to the database, and then show this to your customer. If they can load it without their browser crashing and be able to look through it easily, then continue.

This is more of a case of a customer not understanding what they are asking for than a technical issue.

BTW here's an article on Pagination that may be useful. Do your really think when Google says "About 10,200,000 results for your query" they have a system designed, or ever expect, users to iterate through 10+ million results? What it sounds like you need, besides pagination, is a search engine built on top of your data. Granted, in this instance, simple database indexes may suffice.


My Blog: Down Home Country Coding with Scott Selikoff
Jeanne Boyarsky
author & internet detective
Marshal

Joined: May 26, 2003
Posts: 30894
    
158

I agree that you should be using paging. In addition to Scott's link, see our PaginationOrPaging FAQ page.


[Blog] [JavaRanch FAQ] [How To Ask Questions The Smart Way] [Book Promos]
Blogging on Certs: SCEA Part 1, Part 2 & 3, Core Spring 3, OCAJP, OCPJP beta, TOGAF part 1 and part 2
Sami Ahmad
Greenhorn

Joined: May 18, 2010
Posts: 9
Thank you guys for your help
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: How I can retrive 40 million records from Database?