Need to Unit test an app that submits a form to a webpage

[Jon McDonald]

Hey All,
I'm developing an app (a bot really) that retrieves information from webpages by submitting forms to the page. I'm wondering how can I unit test this feature.
Just to make it clear, I do not wish to unit test the servlet/jsp that the form is being submitted to. Rather I am trying to unit test the bot that will do the submitting. Any help would be appreciated.
Sincerely,
Jon McDonald

[Donal Lynch]

Some issues/hints...
* Network code can be vulnerable to thread timing issues which can make your tests fail intermittently every five or fifty or five hundred times.
* If you don't control the target website, there tends to be a bit of trial and error in finding the correct url and html formats... tests are a good way of being systematic about this trial and error... if a page breaks your bot in trials, you add it to your tests and fix your bot until it passes the tests.
* There's a question of how heavy do you have to make the test framework... if scale wasn't an issue, we might test our cnn.com headline scraping bots by simulating the whole cnn.com website in our test harness.
* Your test code might be more robust and manageable if you test the following areas of functionality separately (assuming that your bot has a structure along these lines...)
- mapping of information request to url
- the http client itself (mapping of url to html page)
- mapping of html page to list of headlines or whatever (html parsing)

[Ilja Preuss]

Originally posted by Jon McDonald:
I'm developing an app (a bot really) that retrieves information from webpages by submitting forms to the page. I'm wondering how can I unit test this feature.

Basically (and generally), divide and conquer: Identify the different steps you need to do and test them separately.
For example, you probably need to parse HTML pages for the information you want. You can develop and test this parser without any thought on sending forms over the net etc.
Did that help?

[Jon McDonald]

Thanks for the info guys!
I think I'll go the divide and conquer route. It should also make for more reusable code that way.
Jon

[Ilja Preuss]

Originally posted by Jon McDonald:
It should also make for more reusable code that way.

Yes!
In fact, that is one of the reasons why many developers nowadays find that taking testing into account from early on (often even before writing the actual production code) makes for better designs: for code to be testable, it needs to be well decoupled.
See http://c2.com/cgi/wiki?TestDrivenDevelopment