Win a copy of Think Java: How to Think Like a Computer Scientist this week in the Java in General forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

How I can retrive 40 million records from Database?

 
Sami Ahmad
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello all,

please I am trying to retrieve 40 million records from Oracle Database, but my JSF page still loading without retrieve anything.

FYI, I use SQL statement to retrieve these data.

So is there any tools help. (hadoop Apache software not supported for Windows Production, only for development)

Thank you in advance.

Sami.

 
Tom Reilly
Rancher
Posts: 618
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Because you are using JSF, I assume you want to display the information to a user on a web page. I'll skip asking why a user would want to see 40 million records. Does your app work when you try to display one record? If you post your code, maybe someone can comment on it.
 
Ulf Dittmer
Rancher
Posts: 42967
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tom Reilly wrote:I'll skip asking why a user would want to see 40 million records.

I won't :-)

As Tom said, so many records should not be handled by the web layer because there's no way for a user to view them. They most likely should not be handled in one batch no matter what: if a record takes just 100 bytes of space then ... well, you do the math :-)
 
Sami Ahmad
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you all, every day my customer sending 300000 messages to his customers, So he want to know what he send before 3 month.
let's say the point not see 40 million records, bcz I can make pagging in my JSF page that will retrieve 500 every search.

but when I am trying to make search with some conditions like From (Date), To (Date), Status, etc.... the page still loading without retrieve anything.

FYI.. the page retrieve 2 million records in 1 min .

Thank you again.
 
Ulf Dittmer
Rancher
Posts: 42967
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Does the search query work if you run it directly against the DB (using the DB's tools)?
 
Sami Ahmad
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you,
I am using Toad, it retrieve 500 row when I make search, I want to know if there tools or software store huge number of data dealing with java language, but NOT hadoop.

Sami .
 
Ulf Dittmer
Rancher
Posts: 42967
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think you should rethink your design. There is no point for the search query to return more records than will be displayed, and IMO 500 records is too many to put on a page meant for human consumption; consider displaying 50 records, or 100 at the most.
 
Scott Selikoff
author
Saloon Keeper
Posts: 4014
18
Eclipse IDE Flex Google Web Toolkit
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think for fun you should create a dummy page that just create 40 million rows, repeated without going to the database, and then show this to your customer. If they can load it without their browser crashing and be able to look through it easily, then continue.

This is more of a case of a customer not understanding what they are asking for than a technical issue.

BTW here's an article on Pagination that may be useful. Do your really think when Google says "About 10,200,000 results for your query" they have a system designed, or ever expect, users to iterate through 10+ million results? What it sounds like you need, besides pagination, is a search engine built on top of your data. Granted, in this instance, simple database indexes may suffice.
 
Jeanne Boyarsky
author & internet detective
Marshal
Posts: 34392
346
Eclipse IDE Java VI Editor
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I agree that you should be using paging. In addition to Scott's link, see our PaginationOrPaging FAQ page.
 
Sami Ahmad
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you guys for your help
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic