• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Deciding on an approach to connect to the web periodically to spawn a web page

 
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
We have a 4 web pages that need to be spawned every few minutes(fixed time interval). The text is then parsed and the program extracts the useful content and stores it into the database.

In the past we implemented threads that run periodically to fetch the data and sleep.

Could you please suggest any other approach that we should follow in implementing these processes or approach or do you think the current approach is good.
 
Marshal
Posts: 28193
95
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I don't understand what you mean by "spawning" a web page. In fact I don't get any picture of what your goal is at all.
 
Kerry Jones
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
An example: lets say parsing a finance document to parse current stock value for a particular company.



Basically the data in these web pages is updated regularly and our program needs to connect to the web page parse the data(retrieve the value of a particular field, as the program already knows the format of the web page) and update the field in database.
 
Bartender
Posts: 9626
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Do you have problems with the current approach? If it is working now, that's probably Good Enough
 
Kerry Jones
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
NO issues as such, But I just wanted to know other developers views as to which approach would have they followed for the same goal.
 
Paul Clapham
Marshal
Posts: 28193
95
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I would have just used a java.util.Timer object which ran at that specified interval and did that specified thing. Keep it simple, and If it ain't broke don't fix it, as Joe Ess suggested.

Unless this task was supposed to be embedded in some existing framework, in which case I would have to look at how to do that sort of thing in the framework.
 
Bartender
Posts: 1952
7
Eclipse IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
So if I understand correctly the program publishes some number of different webpages at a fixed time interval and then proceeds to screen scrape those webpages to read back the information for processing?
That would certainly be a ... novel approach.
Somehow I don't think I'm getting a clear picture here
 
Kerry Jones
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Alright.

Thanks for the input. It is highly appreciated.
 
Paul Clapham
Marshal
Posts: 28193
95
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Jelle Klap:
So if I understand correctly the program publishes some number of different webpages at a fixed time interval and then proceeds to screen scrape those webpages to read back the information for processing?

No, that's what I thought "spawn" meant when I originally read the question. But nothing is being spawned at all. It's just the second part of your description, the code periodically goes out to a web page and scrapes the data it downloads.
 
Kerry Jones
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Jelle Klap, the program does not publish the data.
It only reads the data from the web-pages.
 
Jelle Klap
Bartender
Posts: 1952
7
Eclipse IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Paul Clapham:
No, that's what I thought "spawn" meant when I originally read the question. But nothing is being spawned at all. It's just the second part of your description, the code periodically goes out to a web page and scrapes the data it downloads.



Oh, so it wasn't just me
[ August 18, 2008: Message edited by: Jelle Klap ]
 
reply
    Bookmark Topic Watch Topic
  • New Topic