book Q: bug tracking and 'The List'

[Bridget Kennedy]

Jared and Will:

Congratulations on your publication! Looks like an interesting read.

After perusing the table of contents and samples, I'd like to know a little more about your suggestions for effective bug tracking. Also, regarding your concept of 'The List' of things to-do: Do you touch on strategies for integrating bug tracking, the to-do list, and configuration management?

My team uses Mantis and cvs. Lately, we have been trying to devise procedures (automated or otherwise) to link the two systems.

Thanks - bk

[Jared Richardson]

Congratulations on your publication! Looks like an interesting read.

Thanks very much!

After perusing the table of contents and samples, I'd like to know a little more about your suggestions for effective bug tracking.

The biggest point we make about bug tracking is to actually track the bugs. We know people who've used post-it notes and even today there are prominent consultants/gurus who suggest using 3x5 cards. We think you should use a software product that you can use to generate reports, like all the bugs fixed in the last release or the outstanding bugs in the current product.

It sounds like you guys are already doing this, but many shops aren't. Here is our "Am I Doing This Right" section from the book.


Am I Doing This Right?

If you track issues with any sort of a system, you�re doing great. The
real question is whether you store enough information in the system
and whether anyone uses the system internally. Use these questions
to gauge your success:

Can you generate a list of the top-priority, unaddressed issues?

How about the second-tier issues?

Can you generate a list of last week�s fixes?

Can your system reference the code that fixed the issue?

Do your tech leads use this system to generate to-do lists for development

teams?

Does your technical support team know how to get information

out of the system?

Can you system notify �interested parties� so tech support (and

others) can see when an issue is fixed?

Also, regarding your concept of 'The List' of things to-do: Do you touch on strategies for integrating bug tracking, the to-do list, and configuration management?

No, for Ship It! we're mainly selling the concepts in the first place. We mention linking the products together but don't go into any detail.

For your sitation, Mantis and CVS, have you considered looking at ViewCVS? It lets you add a web link to a file in CVS? If memory serves, it even lets you link to a diff between two revisions... I'm not positive about that. But something like that would let you include (via cut-n-paste) links to the affected files. Sorry the vague suggestion but I haven't used Mantis.

Would do you like it enough to suggest we include in the book's resource page for issue tracking?
[ August 02, 2005: Message edited by: Jared Richardson ]

[Bridget Kennedy]

Thanks for your response, Jared.

Re. ViewCVS: All developers on the team use Eclipse. The Eclipse cvs plugin provides a nice interface for doing diffs and other cvs functions.

...I haven't used Mantis. Would do you like it enough to suggest we include in the book's resource page for issue tracking?

I do like Mantis. It provides a means to achieve most of your
Am I Doing This Right? goals. The interface is highly configurable and full-featured.

My team links cvs checkin with Mantis so the developer doesn't have to repeat documentation generation in two places. Its a combination manual/automated process. The developer must input a specific string at the beginning of the checkin comment - something like "SPCR 57" where 57 is the Mantis issue ID. An automated checkin script keys off this string to add a Mantis note using the cvs comment. A small thing, but every little bit helps.

[Jared Richardson]

Re. ViewCVS: All developers on the team use Eclipse. The Eclipse cvs plugin provides a nice interface for doing diffs and other cvs functions.

ViewCVS isn't for doing diffs or commits. It's a browsing tool for a CVS or Subversion repository. The client is an HTML page. It's a very nice way to browse or link into CVS from a hyperlink.

viewcvs at sourceforge

[Ilja Preuss]

Originally posted by Jared Richardson:
We know people who've used post-it notes and even today there are prominent consultants/gurus who suggest using 3x5 cards. We think you should use a software product that you can use to generate reports, like all the bugs fixed in the last release or the outstanding bugs in the current product.

Well, the gurus I know who suggest using index cards also suggest to work on not having more than a handfull of bugs a year delivered to customers, and fixing them as soon as you become aware of them. I'm not sure how a software product could provide value in such a case...

[Jared Richardson]

Originally posted by Ilja Preuss:


Well, the gurus I know who suggest using index cards also suggest to work on not having more than a handfull of bugs a year delivered to customers, and fixing them as soon as you become aware of them. I'm not sure how a software product could provide value in such a case...

3x5 cards ~might~ work well for a small team for short time (read: consulting engagement), but they simply don't scale, it's too easy to lose one, and you can't generate reports or lists from them except by hand. My employer has nearly 2,000 developers, 300 projects and 15 million lines of code (10m C/C++, 5m Java). If each team used 3x5 cards, what's the best way to gather up the defects for a given product release? Pass around the cards?

How would tech support get a list of known issues? If there are only 3 bugs a year per product, they could memorize them I guess... (kidding!)

Here's a story from Ship It! (page 37). I was there, know Harry and Loyd, and actually found the Post It note.

Consider the case of Harry and Lloyd, intrepid developers, working at
a small software startup. When we met them, they tracked every issue
via Post-it Notes stuck on the wall above their desks. These little pieces
of paper were their only way of tracking bugs, and they assured us
this system worked fine. We still insisted that we use a standard issue
tracking package. We got some resistance and comments about fixing
what isn�t broken, but they humored us and we transitioned into the
new system.

Now this company had a very large customer, one of the reference
accounts, who called in one day and wanted to know when issue X
would be fixed. They said they had reported this issue two releases
back and were tired of waiting patiently. They wanted to know why we
were ignoring them. Harry and Lloyd insisted that they had never heard
of the issue and that the customer was simply confused. We eventually
fixed the issue, and the customer became less angry, but we never got
that account back to the level of �raving fan.�

Six months later, while moving Harry�s desk, we found a Post-it Note
on the floor, right where it had fallen. Written in big, red letters was
IMPORTANT! DO NOT FORGET! followed by the customer�s issue. Once
it fell behind the desk, the �out of sight, out of mind� rule kicked in,
and they forgot the issue. When you use an issue tracking system, you
provide yourself with a safety net that Harry and Lloyd never had.

Your issue tracking system is a bookkeeping detail. You need it to track
what you�ve worked on, what you have and haven�t fixed, and what you
plan to fix. A white board, index cards, or a spiral-bound notebook
might handle your needs for a few months but not for any length of
time, and these certainly don�t scale to the enterprise.

[Ilja Preuss]

Originally posted by Jared Richardson:
3x5 cards ~might~ work well for a small team for short time (read: consulting engagement)

As far as I know, the 3C project (where XP was invented) used index cards for 4 years.

they simply don't scale

I agree that they don't scale to a huge number of bugs - but who wants that, anyway?

it's too easy to lose one

If you made it the top priority of the team to fix bugs as early as they become aware of it, how likely would it be to lose one?

and you can't generate reports or lists from them except by hand.

Yes. Which doesn't sound too much of a problem if you never have more than a handfull at the same time.

My employer has nearly 2,000 developers, 300 projects and 15 million lines of code (10m C/C++, 5m Java). If each team used 3x5 cards, what's the best way to gather up the defects for a given product release? Pass around the cards?

I need more details about the motivation for wanting to have this list to answer your question. (Might well be that for that specific employer a bug database is the best solution. I don't think that from this follows that every project needs one.)

How would tech support get a list of known issues? If there are only 3 bugs a year per product, they could memorize them I guess... (kidding!)

Well, kidding or not, you guessed my answer.

Or they could have a big sheet of paper on the wall listing the current issues, with developers striking out the ones they just fixed.

Here's a story from Ship It! (page 37). I was there, know Harry and Loyd, and actually found the Post It note.

Well, if those customers were used to getting a bugfix a week after reporting the bug, they probably would have asked earlier?

[Jared Richardson]

As far as I know, the 3C project (where XP was invented) used index cards for 4 years.

So what are your thoughts on the 3C project? It was never finished, was cancelled, and the entire shop reverted to traditional software after the consultants left.

Here's a short quote from this page:

There is still debate as to whether or not C3, as it relates to XP, was a success or failure. While it is true that C3 never met its goals, XP did make a doomed system work. The project was incrementally working at the pace it set. The first goal of printing a check in three weeks was met. The next goal of paying 10,000 employees within a year was met. The project was cancelled after four years of using XP, but before XP was threatened with cancellation after just one year. A few questions that are unanswered are "How many user stories were left?" and "How long would those user stories have taken to implement?" It is possible that management simply didn't want to wait that long. It's also possible that XP just couldn't handle the project at all and was going to fail sooner or later. Since the project was cancelled, we'll never know.

DaimlerChrysler is now working on yet another payroll system. They are using traditional methods and software practices. They are even using PeopleSoft. I need to get fresh details but the reports I have now (7/31/04) indicate that they have a big team, the biggest and fastest servers, have worked on it for about two years, and have not paid a single person yet. Compare that to a thin team, obsolete mini computers, and 11 months to actually start paying people with C3. Payroll at a very old, very large company is not as easy as you think which is why C3 was considered a success. --DonWells

Here's a short quote from the marketing material for Extreme Programming Refactored: The Case Against XP.

The authors also examine C3, the first XP project, whose team (most of whom went on to get XP book deals shortly before C3�s cancellation) described themselves as "the best team on the face of the Earth".

My point is not that XP is bad, but I'd love to hear more about ~your~ experience with 3x5s.

Also, I've turned about a decent amount of software over the years, but I've never seen anyone who could actually have a "few" bugs a year. Have you been able to do that?

Or they could have a big sheet of paper on the wall listing the current issues, with developers striking out the ones they just fixed.

And when tech support wants a list of the bugs fixed in the last release? Do you just grab all the sheets of paper included the ones from six months ago that you "archived" in the closet and make a "master" listing?

These sorts of tricks seem like trivial contrived answers to real business problems. Have they ever worked for ~you~? Sure, for an internal payroll project, the XP guys at Chrysler didn't need to have release notes. We ship our software to customers who like to see a list of discovered issues, fixed issues and outstanding issues.

I'm not trying to attack you... you seem like a very smart guy with some great experience... but the 3x5 card issue is (from my point of view) ridiculous. I've never worked in a shop that worked this way and never heard of one outside of vague claims (from RJ).

[Ilja Preuss]

Originally posted by Jared Richardson:
So what are your thoughts on the 3C project?

Well, I wasn't there, but here is what I remember from discussions with people who were:

When C3 was cancelled, it was near to ready to pay the next additional batch of employees.

Things weren't going as smoothly as projected earlier, though. The team was aware of this early in the release cycle, but the project manager indicated that he couldn't tell his bosses this truth. Noone else took the stab on communication this important issue to above. Obviously, when the top managers finally found out, they were upset.

Additionally, top management recently had been totally replaced by new people. There also was the notion that the C3 project actually only was a research project to see wether OOP was something to adopt, that payroll wasn't one of the companies core competencies, and that they should simply buy a ready to use product.

As far as I know, the development team thought that XP worked just great, and that they should use it in the future.

Now, did C3 fail? I'm not sure. The results were interesting enough, though, to get other teams to try it, and it looks like a significant amount of them does so successfully. "My" team isn't doing full XP (yet?), but every technique we introduced has helped us a great deal to produce a better product more effectively and reliably.

By the way, there are some interesting discussions at http://c2.com/cgi/wiki?CthreeProjectTerminated

DaimlerChrysler is now working on yet another payroll system. They are using traditional methods and software practices.

There is an interestin tidbit at the above link: "I did some research and uncovered that the XP winter at DaimlerChrysler was not about the C3 project being successful or not. The gap in XP projects was caused by personalities clashing. XP was not used at DaimlerChrysler for a couple years because of a single man feeling that an enthusiastic XP supporter was disrespectful to him at a single meeting."

Of course you can't be sure how much truth there is behind this story. We probably will never know what exactly happened.

Here's a short quote from the marketing material for Extreme Programming Refactored: The Case Against XP.


quote:
--------------------------------------------------------------------------------
The authors also examine C3, the first XP project, whose team (most of whom went on to get XP book deals shortly before C3�s cancellation) described themselves as "the best team on the face of the Earth".
--------------------------------------------------------------------------------

Well, I can't say that I was very impressed by that book. The authors are very skilled at quoting people out of context, though, no question. <sigh>

My point is not that XP is bad, but I'd love to hear more about ~your~ experience with 3x5s.

My experience with index cards is rather limited, unfortunately. Some people insist in using XPlanner for planning on the current project, which I think has some serious drawbacks. But that's probably something for another thread...

Also, I've turned about a decent amount of software over the years, but I've never seen anyone who could actually have a "few" bugs a year. Have you been able to do that?

Not yet. I'm working on a legacy system that already had a big bug list when I joined the project, and we haven't even yet found out a good way to write system tests for it. It's a long way to go...

Still, if I were in the position to start a fresh new project, I'd start without a bug database, and try to never need one. I've heard of teams being successful in doing so, and I think it's a target worth aiming at.

By the way, as far as I now, the concept actually has its origin in lean manufacturing - the way Toyota builds its cars.

And when tech support wants a list of the bugs fixed in the last release? Do you just grab all the sheets of paper included the ones from six months ago that you "archived" in the closet and make a "master" listing?

These sorts of tricks seem like trivial contrived answers to real business problems. Have they ever worked for ~you~? Sure, for an internal payroll project, the XP guys at Chrysler didn't need to have release notes. We ship our software to customers who like to see a list of discovered issues, fixed issues and outstanding issues.

I'm not at all against release notes! I'm not even against from having a "release notes database". But to me that's quite different from a *bug* database, at least in intent. And as you said, probably not every project needs it.

I'm not trying to attack you...

I noticed that! Thanks for the interesting discussion!

you seem like a very smart guy with some great experience...

Thanks - I can return the compliment. I guess our experience is just significantly different. That's what makes this discussion so interesting, I guess...

but the 3x5 card issue is (from my point of view) ridiculous. I've never worked in a shop that worked this way and never heard of one outside of vague claims (from RJ).

"RJ" = "Ron Jeffries"?

What about http://www.scissor.com/resources/teamroom/ and http://www.xp123.com/xplor/room-gallery/index.shtml ?

[Jared Richardson]

First, thanks for the dialog! I almost didn't post my earlier message... the discussions so far have been so good I didn't want to ruin it! Thanks for the reply!

By the way, there are some interesting discussions at http://c2.com/cgi/wiki?CthreeProjectTerminated

I've never seen that wiki page before! Thanks! It does shed a bit of light on the situation.

I guess our experience is just significantly different. That's what makes this discussion so interesting, I guess...

There wouldn't be much of discussion otherwise, would there?

"RJ" = "Ron Jeffries"?

What about http://www.scissor.com/resources/teamroom/ and http://www.xp123.com/xplor/room-gallery/index.shtml ?

So much for being discrete. heh... Ron is an avid defender of 3x5s in a few of the mailing lists I frequent. He doesn't seem to think that a project with a bug database can possibly succeed.

Thanks for the gallery links too! I hadn't seen those. Many of the pictures do seem to be teams that do projects for other companies. I'm in a company that supports products for years and decades... I wonder how something like 3x5s would attempt to handle that situation?

[Ilja Preuss]

Originally posted by Jared Richardson:
So much for being discrete. heh... Ron is an avid defender of 3x5s in a few of the mailing lists I frequent. He doesn't seem to think that a project with a bug database can possibly succeed.

Well, he probably sometimes sounds a little bit, well, extreme...

But I think he has a point. If we assume that it might be possible to not have more than a few open bugs at the same time, planning for a bug database sounds like overkill.

Actually it *is* my experience that an electronical system easily becomes a place to put a bug and then forget about it, instead of tackling it and reflecting on it.

I'm in a company that supports products for years and decades... I wonder how something like 3x5s would attempt to handle that situation?[/QB]

Remember that the cards are just planning tokens. Once a feature is implemented, it is specified in much more detail by the customers Acceptance Tests. The cards aren't needed any longer.

There are teams whichs release ritual includes burning all implemented story cards...

[Jared Richardson]

quote:
--------------------------------------------------------------------------------
I'm in a company that supports products for years and decades... I wonder how something like 3x5s would attempt to handle that situation?
--------------------------------------------------------------------------------



Remember that the cards are just planning tokens. Once a feature is implemented, it is specified in much more detail by the customers Acceptance Tests. The cards aren't needed any longer.

I meant the 3x5s used for the bugs. Onced the bugs are fixed, do you have any type of artifact that is kept around?

[Tony William]

Jared,

For my team, we have use a spreadsheet for logging all bugs and major events / comments. The log will have a no. assigned. We also have document for each problem fix (w/ ref. to the log no.), problem desc, details, changes and revised source files are included in the doc.

Is this good enough? Any other approaches that you can shared?

[Jared Richardson]

Originally posted by Tony William:
Jared,

For my team, we have use a spreadsheet for logging all bugs and major events / comments. The log will have a no. assigned. We also have document for each problem fix (w/ ref. to the log no.), problem desc, details, changes and revised source files are included in the doc.

Is this good enough? Any other approaches that you can shared?

When at all possible I tend to try to use existing software instead of writing my own... check out the list of packages here and have a look at their pages. Bugzilla for instance is a free program that you can set up and see what their default fields are. But it does sound like you've got it covered pretty well.

[Ilja Preuss]

Originally posted by Jared Richardson:

I meant the 3x5s used for the bugs. Onced the bugs are fixed, do you have any type of artifact that is kept around?

Just to be clear, the project I'm working on *is* using bugzilla. I think it's a "process smell", though.

In "full XP", artefacts that you'd keep would be at least one customer acceptance test showing that the bug has been fixed (and isn't reintroduced), and at least one developer (unit) test showing that the software unit responsible for the bug has been fixed.

Our team also recently introduced a lightweight Release Notes tool to communicate changes to our wide range of customers. Other teams might need other additional tools, but I feel that this gets too specific to the particular project to give general advice.

As I think about it, I'm not at all against having a list of when which bug got fixed, if you get value from it. What I feel is suboptimal is a tool that you mainly use to manage the state of a bug before it gets closed - that's how I feel about a bug database.