Thoughts on test Tools

[Amy Phillips]

Hi,
I am soon to begin a project creating a test management and bug tracking tool. Do any of you have an opinions about testing tools? Are they useful and do they give any real benefits? If any of you have used tools which provide test management or bug tracking features I would be grateful if you could tell me what was useful and what was not.
Thanks
Amy

[Matthew Phillips]

JUnit has completely changed the way I program. I find test-first programming keeps my code clean and generally bug free. I use Ant to run my tests every time I build, so if I introduced any bugs between builds, I find them quickly.

[Amy Phillips]

I am hoping that the re-use of test cases in my system will speed up the test preparation process and the bug tracking will improve the quality does this seem feasible?
I ahev only had experiance of testing at one company so am very interested to hear what others do at other companies.
Thanks

[Ilja Preuss]

Originally posted by Amy Phillips:
I am hoping that the re-use of test cases in my system will speed up the test preparation process

How would that work?

the bug tracking will improve the quality does this seem feasible?

No, bug tracking doesn't increase quality. Low tolerance for bugs increases quality. Possibly your tolerance for bugs should be so low that you don't need a bug tracking tool at all. Seriously.

[Amy Phillips]

ahh but how do you know the quality of your product if you don't have accurate (or any) bug reports? Obviously the level quality won't be directly effected by the reports but it will allow an accurate measure to be made

[Ilja Preuss]

Originally posted by Amy Phillips:
ahh but how do you know the quality of your product if you don't have accurate (or any) bug reports? Obviously the level quality won't be directly effected by the reports but it will allow an accurate measure to be made

Oh, OK - if you mean simply recording a measure (like "bugs found per month"), I wouldn't disagree.
I understood you to mean running a bug database or something similar to collect bugs until they get fixed.

[Amy Phillips]

Ilja,
Thats exactly it but the stats of from the bug database are used to create reports. Its kind of similar to Mercury Interactive's Test Director but with less gloss

[Ilja Preuss]

Originally posted by Amy Phillips:
Thats exactly it but the stats of from the bug database are used to create reports. Its kind of similar to Mercury Interactive's Test Director but with less gloss

I guess I just don't understand why you need a *database* to record one number per month.

[Amy Phillips]

Ilja,
Thats just a feature of the application. Basically the tool is for all project members to stored tests and run tests while also reporting bugs found. At the end reports are generated based on the tests performed and bugs found.
What I am basically wondering is if anyone has used a tool like this or can see that it would be beneficial to use one? I am trying to extend this whole bug database thing to be a bit more functional.
The system will be quite generic so that you can perform different types of test i.e. system, unit, acceptance etc and the testers just create the tests and then use them from a library.

[Ilja Preuss]

Originally posted by Amy Phillips:
Thats just a feature of the application. Basically the tool is for all project members to stored tests and run tests while also reporting bugs found. At the end reports are generated based on the tests performed and bugs found.

So the "bug database" is for simply storing test failures (in contrast to bugs found in the field by users)?

What I am basically wondering is if anyone has used a tool like this or can see that it would be beneficial to use one? I am trying to extend this whole bug database thing to be a bit more functional.

Well, if you really mean to measure "number of failing tests per time period", I would see one problem - there are two causes for the tests not failing: a high quality system or low quality tests. So this measure is probably inadequate for improving system quality.

The system will be quite generic so that you can perform different types of test i.e. system, unit, acceptance etc and the testers just create the tests and then use them from a library.

What would be the advantage over putting the tests into the general source code repository?

[Amy Phillips]

I agree with your point about quality so maybe its not quality I mean but the whole project communication thing. How can a project manager know when a project will be completed without reports?
I also think your point about storing unit tests seperatly is importatnt - I believe most tests performed by the testers are system interactiosn to see if the system works correctly. However programmers test for programming bugs. Ok they are very different but the idea with a system like this would be to bring the programmers and testers together therefore it is important the testers can see the unit tests and the programmers can see the acceptance tests etc.
I think that the big problem with programming and testing is that the two teams work so far apart. A system like this would hopefully bridge the gap.
What do you think?

[Ilja Preuss]

Originally posted by Amy Phillips:
I agree with your point about quality so maybe its not quality I mean but the whole project communication thing.

With all due respect, I don't see how a database would improve communication - for that, you need to communicate more, not to store more data...

How can a project manager know when a project will be completed without reports?

Put a big chart of #features_done vs. #features_to_be_done over time on a white board. That should give him some hints. No database needed.

Ok they are very different but the idea with a system like this would be to bring the programmers and testers together therefore it is important the testers can see the unit tests and the programmers can see the acceptance tests etc.

Why don't you just put both unit and acceptance tests into the code repository and give both testers and programmers full access to it?

I think that the big problem with programming and testing is that the two teams work so far apart.

Yes. That is most often because they *are* far apart - both in time and space. In my opinion the best solution would probably be to bring them together - put them into one room and let them collaborate.
I don't think that a tool will have much influence.

What do you think?

Sorry if this seems overly destructive.
We are all developers, and so it's only natural to search for technical solutions to our problems. Annoyingly, the biggest problems in our industry aren't technical in nature, but social - and often, warm, social (and simple) solutions are much better for solving them than the cold, technical (and mind-bogglingly complicated) solutions we so much like to tamper with...

[Richard Jensen]

Originally posted by Amy Phillips:
I am soon to begin a project creating a test management and bug tracking tool.

I would like to understand more of the nature of your project. I've read a lot of the responses Ilja gave, but I don't understand the motivation for the project.
Do any of these come close?

Work--Product

Work--internal tool

Personal--Product

Personal--learning

And the reason that I ask is that if it is Work/Internal are you sure this is the best build/buy decision you can make?
Have you looked at what JUnit/Ant can put out?
I'm not sure what eclipse uses, but have you seen what they have in terms of automated report status accessible via the browser? Go to http://www.eclipse.org/downloads/ (choose an appropriate mirror) and click on the link by the latest 3.0 Stream Nightly Build. (It was N20030624 when I composed this message).
You see big X'd boxes for the things that aren't building. You can follow the link from one of these and see what the tests performed vs. errors/failures is and drill down even further.
So, what are you envisioning that is different/solves a different need/enables better communication?

[ARS Kumar]

Hi Amy
Are you planning to work in a project that is indent to create something like this ?? As you know http://bugzilla.mozilla.org/ is free bug tracking tool. Soffront tool has a backend database and a very strong reporting facility. I do understand the need and importance of reporting for project and quality managemnt process. I am just wondering what will be the motive to start a brand new product of this kind if there is enough market leaders out there ? If you have time please explain.
Thanks for your time
[ June 24, 2003: Message edited by: ARS Kumar ]

[ARS Kumar]

Hi Amy
I actually got the answer from the message you put in the other thread
Anyway good luck in creating a new bug tracking tool !

[Amy Phillips]

Hi,
This project is a university project where I need to build a system. I thought it would be nice to build a solution rather than just a game or something. However you are of course right in the fact that there are already loads of bug tracking tools available this was why I wanted to make it a little different by adding the test management part. Are there any test management tools avaliable on the web and do any of them also contain a bug tracking tool?
Ilja, I think your points are valid but I do know that a lot of people think this sort of tool is worth while for example where there are location problems or when the project is particulary large. A manager doesn't want a white board of notes they want a concise report or preferably a graph to quickly see whats going in.

[Frank Carver]

Have you looked at http://fit.c2.com ?
It's a deceptively simple idea which really helps to integrate testing, bug tracking and reporting. Don't get put off that it appears to be just some web site, there's some very clever stuff going on there.

[Amy Phillips]

Frank,
Thank you very much that site is excellent!
I think that this thread has given me another idea - two standalone systems which could be integrated? One could be the test management and the other the bug tracking then you can choose which to use.
Sound better?
Amy

[Christian Hargraves]

Amy,
I have been working a tool that you seem to be talking about. Jameleon is just a start to a test case management tool. Basically, the idea is to break an application up into features, code those features, document those features and then create test cases based on those features. In other words, the documentation and test scripts are generated from the order of the features you group together.
Currently, the test case management peice isn't done. I am, however, working on that peice.
I was thinking about implementing a bug tracking plugin interface used to communicate with an external bug tracker.
Check out Jameleon
and tell me if that isn't something you were thinking about. I am looking for contributors to help me out on this.