• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Read Data from webpage

 
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
hi 2 all

i want to know how can i read the data from a webpage which continuously refreshes itself..
i tried but i got html source code of page that time..
 
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
What differentiates "the data" from "the html source code"? They're both the same thing, no? If you want updated HTML, you'll have to read the page again, but you may have problems if the page uses AJAX to refresh parts of itself - that's not easily amenable to screen scraping.
 
Kuldeep singh ranbanka Rathore
Greenhorn
Posts: 13
Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Sir,
for example if we talk about any website who provides you live score(which automatically gets updated) of any game, then how can read that data(say score)
 
Ulf Dittmer
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
My premier tool of choice for that would be a library like HtmlUnit (or jWebUnit, which is built on top of it). But again, if the page refreshes, then those libraries need to reload the page - that doesn't happen automatically. And if the live update happens through AJAX, all bets may be off.
 
author
Posts: 23951
142
jQuery Eclipse IDE Firefox Browser VI Editor C++ Chrome Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Kuldeep singh ranbanka Rathore wrote:Sir,
for example if we talk about any website who provides you live score(which automatically gets updated) of any game, then how can read that data(say score)



There is no one answer -- it depends on what the page is doing. And how it is getting the information from the server.

All the URL does is get the initial information for the page. This page can load tons of stuff, including javascript, flex, sl, java, xslt, etc. etc. which make calls to the server to get more information. It may also do tons of processing too, which means if you want the "score" displayed on the page, you need to know how the page application loads the information, and how it processes it for display.

Henry
 
Ranch Hand
Posts: 245
2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
red this again:

Ulf Dittmer wrote:but you may have problems if the page uses AJAX to refresh parts of itself - that's not easily amenable to screen scraping.



my opinion, pretty futile to attempt to do that, unless prowider gives you access to some web service.
 
Kuldeep singh ranbanka Rathore
Greenhorn
Posts: 13
Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
So how can i know that how any website is refreshing...? any tutorial regarding this?
thanks in advance..
 
Ulf Dittmer
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Miran Cvenkel wrote:my opinion, pretty futile to attempt to do that, unless prowider gives you access to some web service.


Not necessarily. HtmlUnit works fine with many common JavaScript libraries like Prototype, jQuery, Ext JS, Dojo, GWT etc. Someone not averse to tinkering might just get it to work.
 
Ulf Dittmer
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Kuldeep singh ranbanka Rathore wrote:So how can i know that how any website is refreshing...?


By analyzing its source code. It might be as simple a having a Meta refresh header in which case simple reloading would work. Or it might be using JavaScript, in which case it gets trickier, and you may have to analyze the actual HTTP traffic that goes over the wire.
 
Kuldeep singh ranbanka Rathore
Greenhorn
Posts: 13
Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
can you tell me about the following website?webpage
thanks for help you all........
 
Ulf Dittmer
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If you want to get this up and running then you will need to become knowledgeable about HTTP and HTML; no better way to start that than to do it yourself
 
Kuldeep singh ranbanka Rathore
Greenhorn
Posts: 13
Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
actually all i want to do is that i want to make an application which can fetch and show live scores from the website providing live scores.
 
Ulf Dittmer
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Make sure that what you're doing is legal - most web content has a copyright that prohibits republishing.
 
Kuldeep singh ranbanka Rathore
Greenhorn
Posts: 13
Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am a student and trying to do that only for testing purpose not commertial. By the way, thanks for advice
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
For understanding how a web page is constructed, I suggest the Firefox browser with plugins like Firebug that let you examine all the components.

Given the amount of money involved in sports data licensing, you may find that the designers have made it really difficult to retrieve only the scores.

Way back (1996) when Java first got started I helped make a "Instant Sports" applet that animated baseball games live while reporting scores and status of other major league games. When MLB (Major League Baseball) got an idea of how much they could get for this stuff we were priced and lawyered out.

Bill
 
Ranch Hand
Posts: 405
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

You can use com.gargoylesoftware.htmlunit.WebClient for this.



 
Ulf Dittmer
Rancher
Posts: 43081
77
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

You can use com.gargoylesoftware.htmlunit.WebClient for this.


...which is what I suggested in my second post.
 
Kuldeep singh ranbanka Rathore
Greenhorn
Posts: 13
Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
thanks ujjwal and ulf.....
 
reply
    Bookmark Topic Watch Topic
  • New Topic