• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

When to use Screen Scraper and Object mapping?

 
Ranch Hand
Posts: 1312
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
When to use Screen Scraper and Object mapping?

thank you.
 
Ranch Hand
Posts: 995
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
What are these?

./pope
 
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You generally need to resort to screen scraping when there is no programmatic API provided to access the data you need to access from a legacy application. For example, someone might've developed a web application on top of a home-grown flat file database and didn't provide a web services API or anything similar. If you really need access to the data inside that flat file database, it might be easiest to scrape the screen on that web application.

For object mapping (I assume you mean object-relational mapping?), we have a dedicated forum here at the 'ranch but, in short, you need to do object-relational mapping when you want to represent a relational data model as an object model.
 
Ranch Hand
Posts: 8945
Firefox Browser Spring Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Lasse
What is Screen scraping? I havent understood.
 
Alexandru Popescu
Ranch Hand
Posts: 995
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I think that a screen scrapper is a tool that investigates the information in the presentation format (for example: extract the information out of a web page) without having access at the real application.

./pope
 
Ranch Hand
Posts: 188
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
A screen scraper is an app that literally "reads" the presentation layer of some other app and creates some representation of the info on the scraped screen. For example, the current project I'm working on uses a screen scraper to read fortran-based green screens, takes the info from them and puts it in an XML file, which we in turn parse into transfer objects that are used to populate JSPs. In our case, it was a quick way to reuse the business logic in the legacy code instead of reimplementing it in Java.
 
Alexandru Popescu
Ranch Hand
Posts: 995
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Eric Fletcher:
For example, the current project I'm working on uses a screen scraper to read fortran-based green screens, takes the info from them and puts it in an XML file, which we in turn parse into transfer objects that are used to populate JSPs. In our case, it was a quick way to reuse the business logic in the legacy code instead of reimplementing it in Java.



What about the app performance? Extracting info, than creating XML, deserializing XML to object, JSP seems to me quite a long way for a request to go be back with its response.

./pope
 
Eric Fletcher
Ranch Hand
Posts: 188
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Ali Pope:


What about the app performance? Extracting info, than creating XML, deserializing XML to object, JSP seems to me quite a long way for a request to go be back with its response.

./pope


Performance isn't bad, actually. Since the underlying data model(DB2 on AS400) had been optimized for calls from the exisiting business logic, the scraper performed pretty well vs direct database access via JDBC. It was quicker than us coming in and learning the data model and business rules from scratch, we had the XML schema so we just needed to implement the parsing. We are migrating to direct access eventually, but they wanted something up and running yesterday, so...
 
Alexandru Popescu
Ranch Hand
Posts: 995
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Great to hear this. But still I am asking way to go XML - Objects - JSP and not trying to cut out one of those layers (either parse directly to Objects or process XML with XSLTs)?

./pope
 
Eric Fletcher
Ranch Hand
Posts: 188
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Ali Pope:
Great to hear this. But still I am asking way to go XML - Objects - JSP and not trying to cut out one of those layers (either parse directly to Objects or process XML with XSLTs)?

./pope



Couple of reasons. First we knew we would eventually be going the route of direct access to the database so we knew we would need the transfer objects built eventually. Since that was the direction in which we were heading, XSLT wasn't really considered. We used a typical DAO layer and a factory to abstract the data source. Whether the TO's are populated by the scraper or by direct DB access is configured by a factory property and our business objects that call the DAO methods are none the wiser.

As far as parsing directly to objects, we are using a third party scraper that returns an XML file representing the structure of the scraped view, so that wasn't an option.
 
Alexandru Popescu
Ranch Hand
Posts: 995
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Congrats Eric! Very elegant and powerful thinking and solution!

./pope
 
Eric Fletcher
Ranch Hand
Posts: 188
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Ali, I appreciate the kind words!



Cheers!
E
 
(instanceof Sidekick)
Posts: 8791
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Screen scraping was called "robotic typing" by a few vendors. I kinda liked that. We do it on CICS green screens that are visible to the user. We send keystrokes to navigate through the first part of a transaction to a screen where they can finish it. With a good framework in place we can provide new keystroke scripts at very low cost. We read (scrape) screen data much less often.

Vendors like Attachmate have highly automated screen scraping and put it behind standard interfaces (as Eric did with his DAOs) to turn mainframe apps into web pages or data sources.

A decade ago the standard API for this was called HLLAPI - High Level Language API for 3270. Atachmate has wandered away from that to a variety of proprietary interfaces.
 
Don't sweat petty things, or pet sweaty things. But cuddle this tiny ad:
a bit of art, as a gift, that will fit in a stocking
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic