aspose file tools*
The moose likes Testing and the fly likes Automated Web Reading & Extraction! Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Spring in Action this week in the Spring forum!
JavaRanch » Java Forums » Engineering » Testing
Bookmark "Automated Web Reading & Extraction!" Watch "Automated Web Reading & Extraction!" New topic
Author

Automated Web Reading & Extraction!

anand phulwani
Ranch Hand

Joined: Sep 10, 2005
Posts: 242
Dear Ranchers,

I have a website to which i wish to log in and check my usage details,and add the records to my database,for this i have to open the website's homepage,fill in my UserID and Password programatically,and then the usage details hyperlink clicked,then i wish to extract these details and add it to my database.This whole process will be done programatically without any human interference.A rancher adviced me HtmlUnit,i have gone through the website and the api,its a somewhat similar solution what i wanted except that i am facing these problems.

1)I am not able to find out how can i render the page,as i want to see the HTML page.As far as i think there could be a parser which could take HtmlUnit object and display its contents as it happens in a browser.

2)I also wish to see the filling of the UserID,Password Details filled,and when i fire the clickevent() on the submit button.i wish to observe the button clicked.

This is somewhat similar to Visual Basic Inet Control on which my friend has developed the same,and i wish to do it now in Java.
Hoping someone helps.
[ February 07, 2007: Message edited by: anand phulwani ]

Thanks and Regards, Anand
SCJP 5.0 310-055 73%, SCWCD 1.4 310-081 78%, IBM DB2 9 Fundamentals 000-730 62%
Jeanne Boyarsky
author & internet detective
Marshal

Joined: May 26, 2003
Posts: 30764
    
156

Anand,
HTMLUnit is good for the programmatic part. However as you've noticed, you can't see the output.

How about using a record/playback tool that operates out of the browser. For example, Selenium is a Firefox plugin. Any of the other record/playback tools for testing would accomplish the same thing.


[Blog] [JavaRanch FAQ] [How To Ask Questions The Smart Way] [Book Promos]
Blogging on Certs: SCEA Part 1, Part 2 & 3, Core Spring 3, OCAJP, OCPJP beta, TOGAF part 1 and part 2
anand phulwani
Ranch Hand

Joined: Sep 10, 2005
Posts: 242
Dear Jeanne,

Thanks for your reply,really grateful to you from my side,
i will look over selenium and do some practicals of it today itself,
and then repost on this thread,but still thanks a lot.

anand phulwani
Ranch Hand

Joined: Sep 10, 2005
Posts: 242
Dear Jeanne,

Thanks a lot,it worked as i wanted till 99% ,i just want to know that can i integrate the browser into my java application itself,because i dont wish the user to see anything out of the application,and can the customized frames of selenium be removed out of the explorer window.

But keeping the problems aside,
Many Many Thanks to you.
Jeanne Boyarsky
author & internet detective
Marshal

Joined: May 26, 2003
Posts: 30764
    
156

2)I also wish to see the filling of the UserID,Password Details filled,and when i fire the clickevent() on the submit button.i wish to observe the button clicked.

You can launch this through a command line using chrome://. See the Selenium documentation for the exact syntax of the URL. And you can run command line programs in Java through Runtime.exec().

I'm pretty sure you can't get rid of the frames entirely as Selenium uses them. You can shrink them so small that the user can't see them though. Take a look at the selenium/TestRunner.html file. It should contain a frameset. (I say should because I don't have selenium on my home machine to check.) Edit this file to set the frames to something tiny.
anand phulwani
Ranch Hand

Joined: Sep 10, 2005
Posts: 242
Dear Jeanne,

Thanks a lot,yes that can be done if i do some research,and yes i know about Runtime.exec(),i can start the server by this command,but what i am thinking is i dont want to display the things in the internet explorer itself,i want to integrate just the browser display part into my swing application,i hope this could be done and by this my goal would be achieved,Jeanne you helped a lot and you really deserve
,three cheers.
Jeanne Boyarsky
author & internet detective
Marshal

Joined: May 26, 2003
Posts: 30764
    
156

Thanks Anand!

For the Swing part, you may want to start a new thread in Swing/AWT/etc. Feel free to link to this one so there is some context.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Automated Web Reading & Extraction!