defect tracking column for SW Test & Perf Mag

[Geoff Koch]

I'm writing a column on defect tracking best practices for Software Test & Performance magazine and am wondering if I might pose a few questions here? Like last time, I'm looking for anecdotes, quips, stories, etc., from folks willing to go "on the record." One question I'm curious about: with the rise of agile practices and software as a service distribution models, is defect tracking destined to become less important than it is today? (Because with the former, fewer bugs are produced in the first place and with the later, bugs are dealt with more quickly [than would be the case with traditional shrink-wrapped release cycles.]) Are we headed back to a time when we can track defects on a spreadsheet or even on an office white board? Or am I all wet on this point? Thanks, all, for any help you can provide. Deadline for this one if Friday, May 12.
Best regards,

[Jeanne Boyarsky]

I'm moving this to our testing forum as it seems to be more about testing/defect tracking than tools.

[Jeanne Boyarsky]

We currently use an Excel spreadsheet for defects noted by the developers. The idea is that once a developer declares a feature as "done", anyone (including the initial developer) logs any defects found with that feature. That way we find things while working on other features and through testing. Agile methodologies just move the defect finding phase up in the process.

It's useful to go back through the older defects to find patterns. We've written some "unit tests" that use reflection to try to identify some of the recurring families of defects and stop them from occuring. For example, we now have tests for forgetting to log a field when it is added to an object and for forgetting aspects of usability/Section 508 in a JSP.

As we get more data (6 applications over 4 years) from the same team, we are noticing that the spreadsheet model is becoming harder to use. There's just too much data to find useful patterns in any methodical way. So we are looking towards an actual defect tracking tool.

I think the spreadsheet model still works well for small projects. I recently started one for the common department build project. Since there are only one or two developers, we started out without one. But it helps having a shared defect list and not just keep the information in our heads. The spreadsheet also helps with noticing trends that keep getting reported as defects, but are in fact working "correctly."

To sum this up, I think the history of defects is more important than just the defects themselves. Over time, the spreadsheet model is poor for history. And the whiteboard system is even worse.

Almost forgot to add the contact information you usually ask for:
I am a Java developer for a bank in New York city

[added contact info]
[ May 08, 2006: Message edited by: Jeanne Boyarsky ]

[Geoff Koch]

Thanks very much, Jeanne. Even if agile programming generates the same number of defects for a given bit of programming, doesn�t it mitigate the need for a complex tool? If agility is sort of marked by a �see defect, fix defect� approach, then there would seem to be no need for big complex defect tracking repositories of any kind. Am I hopelessly na�ve? (It�s a distinct possibility.)

Do you have a thought as to how to know when you�re past the tipping point, after which you need a tool instead of a spreadsheet? I imagine it�s very situation-dependent. And when you say you�re considering a tool, do you think it will be a freeware program or something you pay for? The vendors out there say yours is just the kind of situation that a good proprietary defect tracking tool is designed to handle. However, I know there are lots of freeware options out there, too.

One challenge in writing this column, I realize, is that there�s a common theme for many of these topics. On the one hand, many folks say that most projects, in the hands of skilled developers, can be handled with merely a text editor, a spreadsheet and a few decent whiteboards. Of course, others, especially but not exclusively all the vendors, say that tools that hide some complexity are paramount, and so on. I guess my job is to ask the right questions to get beyond this basic impasse.

Thanks again for your response on this one and for directing me to the more appropriate forum. Sorry I posted in a less-than-perfect spot the first time around.

-Geoff

[Jeanne Boyarsky]

Originally posted by Geoff Koch:
Even if agile programming generates the same number of defects for a given bit of programming, doesn�t it mitigate the need for a complex tool?

Why does a tool need to be complex? I could argue that using a tool is less complex. We have to take the time to set up the optimal fields in a spreadsheet or come up with a whiteboard technique. This is equivalent to using a tool in preparation effort. Tools often come with sample formats that you can use out of the box.

Originally posted by Geoff Koch:
If agility is sort of marked by a �see defect, fix defect� approach, then there would seem to be no need for big complex defect tracking repositories of any kind. Am I hopelessly na�ve? (It�s a distinct possibility.)

Agility has disciplne too though; it isn't a free for all. If someone sees a defect, do they drop what they are doing and immediately fix it. Maybe for a trivial one. But what if the defect takes 3 hours to fix. As soon as someone else might be fixing the defect or the defect finder defers the defect, I think it should be recorded. Otherwise, it could be lost track of. You use this "complex" word again. Why is a tool/respository inherently complex? Incidentally, we don't say that we won't use a source code repository because we are agile. Why should defect tracking be different?

Do you have a thought as to how to know when you�re past the tipping point, after which you need a tool instead of a spreadsheet? I imagine it�s very situation-dependent. And when you say you�re considering a tool, do you think it will be a freeware program or something you pay for? The vendors out there say yours is just the kind of situation that a good proprietary defect tracking tool is designed to handle. However, I know there are lots of freeware options out there, too. [/QB]

It is definitely very situation dependent. I think once you start to feel pain with the spreadsheet is the time to start looking for a tool. We used a spreadsheet without any difficulties until we got past the 4 applications and 10 developers stage. After that, things got awkward.

And on my other team, the small project, we started using a spreadsheet when its predecessor (paper) got painful. As you can guess, this happened fairly quickly. I think that project will outgrow the spreadsheet once it gets to around 50 reported defects (some of which may not be defects.) This project may hit the limit sooner because it is even more important to be able to search history.

I think we will use a freeware tool. Right now, we need "just a little more." Maybe in the future, we will revisit this if we shortcomings in the free tools.

One challenge in writing this column, I realize, is that there�s a common theme for many of these topics. On the one hand, many folks say that most projects, in the hands of skilled developers, can be handled with merely a text editor, a spreadsheet and a few decent whiteboards. Of course, others, especially but not exclusively all the vendors, say that tools that hide some complexity are paramount, and so on. I guess my job is to ask the right questions to get beyond this basic impasse. [/QB]

I think I'm in the middle. I think things can be handled quite well with a spreadsheet/whiteboard. (not so much for the text editory) But I also find trend analysis to be useful. That may be because I am on a team/organization where people stay longer than average (in the programming industry) so we have more time to learn about our team traits and patterns. But it also shows were new people make the same mistakes and then we can put those in our "Welcome" document.

Thanks again for your response on this one and for directing me to the more appropriate forum. Sorry I posted in a less-than-perfect spot the first time around.[/QB]

No problem. That's what moderators are here for!

[Geoff Koch]

Thanks so much, Jeanne! I really appreciate your answers, particularly about agile processes. Take care! (Do you have any tips on getting others in the forum to respond if/when I post questions for future columns?)

[Lasse Koskela]

Hi Geoff,

Originally posted by Geoff Koch:
I'm writing a column on defect tracking best practices for Software Test & Performance magazine and am wondering if I might pose a few questions here? Like last time, I'm looking for anecdotes, quips, stories, etc., from folks willing to go "on the record."

I meant to post some thoughts already several days ago but, as usual, the busy-bug got me. Anyway, I hope this isn't too little too late for you. At least, I hope this will generate some discussion.

I'm an XP/Scrum/Agile coach based in Helsinki, Finland and working for a consultancy named Reaktor Innovations.

Originally posted by Geoff Koch:
One question I'm curious about: with the rise of agile practices and software as a service distribution models, is defect tracking destined to become less important than it is today? (Because with the former, fewer bugs are produced in the first place and with the later, bugs are dealt with more quickly [than would be the case with traditional shrink-wrapped release cycles.])

Defect tracking is not destined to become less important than it is today. What we're seeing happening with successful adoption of agile methods is that the smaller volume of defects allows us to use simpler tools to keep track of them. I've seen several projects during the past year having gone from using a rather heavy software tool for defect tracking to tacking the defects on a whiteboard with stickies along with the features to implement in that iteration, and only use the defect tracking system as a glorified database.

Putting defects on the team's task whiteboard did wonders in one project where defects had previously been effectively hidden from the developers and, in practice, had to be assigned by the project manager in order to get them fixed. After adding all defects above certain priority/severity threshold to the whiteboard, people couldn't help but understand that the defects really need to be fixed. One day, I realized that I was witnessing a competition between two developers of who fixes more bugs. That had never happened when the defects were just entries in an obscure tool.

Visible information radiators are excellent tools. While software is good at storing information, information radiators like whiteboards are good at making people act on that information.

Originally posted by Geoff Koch:
Are we headed back to a time when we can track defects on a spreadsheet or even on an office white board? Or am I all wet on this point?

In a way, yes. There will always be projects where the spreadsheet or whiteboard won't do. Having said that, I do think we'll see more and more large enterprises giving up on their irrational policies of mandating the use of heavyweight tools for projects that just need a whiteboard.

I hope you find this useful. And I hope you're not working all night to make the deadline

[Jeanne Boyarsky]

Originally posted by Geoff Koch:
Thanks so much, Jeanne! I really appreciate your answers, particularly about agile processes. Take care! (Do you have any tips on getting others in the forum to respond if/when I post questions for future columns?)

I think part of it is time as Lasse noted. Usually there are a few people who respond to these things.

Another approach is to ask a question in your subject. Maybe "how do you use defect tracking in agile dev?". Rereading the subject, this sounds a bit like a link to an article.

[Lasse Koskela]

Re-reading my post above, I think it needs some clarification.

Originally posted by Lasse Koskela:
Putting defects on the team's task whiteboard did wonders in one project where defects had previously been effectively hidden from the developers and, in practice, had to be assigned by the project manager in order to get them fixed. After adding all defects above certain priority/severity threshold to the whiteboard, people couldn't help but understand that the defects really need to be fixed. One day, I realized that I was witnessing a competition between two developers of who fixes more bugs. That had never happened when the defects were just entries in an obscure tool.

The project I'm referring to wasn't what I'd call an agile project. It had been going on with a "Big 5" consultancy-driven waterfall process for 2 or 3 years before they got a smart architect and a project manager with balls. After that, they started moving from a one release per year cycle towards a release every 3 months. I'm not sure whether they ever managed to get even shorter releases. The customer would've wanted shorter releases just like the development team would've, but there was some bureaucracy in between that wanted to keep things slow--just so it's easier for them to keep track of the releases.

In other words, this is happening not just for agile projects but projects that are transforming towards being more agile.

[Ilja Preuss]

Originally posted by Lasse Koskela:
While software is good at storing information, information radiators like whiteboards are good at making people act on that information.

Great line! Expect me to steal it...

[Ilja Preuss]

Ron Jeffries has a related article on his website: http://www.xprogramming.com/blog/Page.aspx?display=PlanningSoftware

[Geoff Koch]

All,
I'm remiss in following up and saying thanks for the help here and for your comments. They were wonderful, as always, in helping me understand the topic and at least a few of them did make it into the column. I'll certainly be back with questions about my next column, if that's ok. The topic is testing with Eclipse.
Thanks again!
Geoff