Hi J.B.: I'm not an expert at Agile methods, so please take my reply with a grain of salt. I would like to learn from you about Agile Customer Testing (and Acceptance Testing) so feel free to reply.
Bret Petticord introduced me to Agile Testing a couple of years ago. It seemed very well thought-out and something that would challenge engineers to think about testing first. UGOT is different from Agile Testing in these ways:
1) Agile stresses testing-first where a unit/functional test is built before and engineer writes the application software. UGOT says "test first" is a good idea but it is optional. UGOT is not nearly as militant as Agile sometimes comes across.
2) UGOT provides a decision maker with actionable knowledge to decide when a service is ready for production. UGOT provides:
- Functional test to check for regression and successful operation.
- Scalability testing for capacity planning. Scalability is stated as a function of throughput measured at the consumer/client in requests-per-second as the number of concurrent consumers/clients increases. This shows that the system has enough capacity to serve forecasted levels of users. (See
http://www.pushtotest.com/Docs/howto/onscalability2.html to see how I have been plotting this for our customers. I'm open to feedback/criticism.)
- Performance testing to make sure the service meets user expectations for response times. A 3 tall mocha latte's a day person isn't going to wait more than 5 seconds for their email client to check for new messages.
3) Bret described "coaching tests, a way of thinking about Acceptance tests" that turn user stories into tests. UGOT identifies archetypal users by their behavior when using the service. By understanding the archetypal users behavior, we can then write test agent code that drives the service, as the archetypal user will. For example, the following are two archetypes for an upcoming test of a payroll service:
Payroll Data Entry Clerk (Irene)
Irene works for a paper supply company in Milwaukee, Wisconsin. She is 26, engaged to be married next Spring, an accomplished long distance runner. Irene is response for managing the company and employee information for the company payroll. On a daily basis she gets questions from employees and contractors about their tax withholding. For example, how many dependents do I have on my W2? The company makes a payroll every 2 weeks. Irene's work effort greatly increases as the next payroll date approaches.
Payroll Approver (Maggie)
Maggie works for the same paper supply company as Irene. She manages the in-house bookkeeping staff of 3 clerks, including Irene. Maggie has been with the company for the past 12 years as the financial services manager. She has two children, her husband is a small business owner, and she loves to travel. Maggie is the companies' final check that payroll information is correct and that the calculated taxes in each payroll are correct. Maggie routinely updates company and employee information.
The user archetypes provide a common, down-to-earth, way of discussing typical usage of a service among developers, QA techs, and IT managers. UGOT turns the archetypes into functional tests. This can be done in any framework: jUnit, Load Runner, jMeter, Java, Jython, etc. (Of course, my preference is Jython and Java in TestMaker.)
With the right framework these functional tests are run multiple times and concurrently to test the system for scalability and performance. The same functional tests can be run periodically as a Quality of Service monitor.
Java Testing and Design shows the reason why building archetypes is important to a business, how to go about doing it, and then how to repurpose them between developers, QA technicians, and IT managers.
-Frank