Traceability

[Srinivas Krishnan]

This is a question for the authors, how do you think agile methodologies handle traceability from 1 phase to another, with multiple iterations?

In my experience traceability place a key factor in developing a top notch design that can survive change. Can you provide any insight on this aspect ?

[Will Gwaltney]

Originally posted by Shrin Krishnan:
Can you provide any insight on this aspect ?

In short, no. I have to admit I'm not familiar with the term "traceability" with respect to software, but I suspect that it's defined in one of the more formal software development processes. Can you give me (and the group) some information on what it means and how it's used?

[Srinivas Krishnan]

Traceability in the sense I am using it has to do with maintaining a coherent flow from artifact to artifact as we move from thru various phases of design analysis->design->code etc. Hence, if we ever chose one use case, we can go from analysis to design easily, and in some case design to analysis.

Usually when we iterate at some specific stage the traceability is broken, as the changes are not reflected in the all the connected artifacts. So, I think the analogy is of a tree with various nodes and all nodes getting updated when design changes or iterations take place.

Mostly people have talked about using UML diagrams to maintain this semantic relationship. We have talked about using a layered approach using patterns to constrain the scope and preserve traceability. So, analysis patterns map to design patterns to coding idioms etc. The multiplicity at various level transitions has also been taken into consideration using PUGH approach.

I would be more than happy to send you links to paper we have published with some preliminary work, to get your comments on them.

The main stumbling block is to make it lightweight enough to be used in the industry regularly. But we are making tools like Rations Rose to solve this.

What is your opinion on this, in your experience how is design kept coherent with change and iterations.

[Scott Ambler]

An agile projects, traceability isn't as big of an issue because we travel light. In particular, if you adopt the practicesingle source information you greatly reduce the need to trace.

You can also reduce traceability needs through common naming conventions.

Traceability is often considered to be a process smell, indicating that you're likely documenting far too much and/or are following bureaucratic processes which could easily be trimmed down. Traceability should really be built into your toolset and handled automatically. If you need to use a separate traceability tool such as Caliber RM or Requisite Pro then you're likely in trouble because your trace matrix will get out of sync with your other artifacts.

- Scott

[Will Gwaltney]

Originally posted by Shrin Krishnan:
What is your opinion on this, in your experience how is design kept coherent with change and iterations.

First off I'll deal with the "normal" agile project, where traceability as you describe above is not specified in the contract. I don't think traceability is going to be very important here. If your customer is involved during development and is providing substantive feedback during each iteration, the end result is going to look very different than the initial design. This is a Good Thing, because usually the initial design has little to do with what the customer actually wants/needs. Traceability isn't too important because nobody's really going to care in a year or two what the product looked like during any given iteration. Usually the record of code changes captured in your source code management system is going to suffice. In our book we also discuss sending out Change Notifications when code is checked in, to let interested parties know when a change is made. Actively sending these notifications out (via e-mail or the like) works better than relying on the group to peruse the SCM on a regular basis.

For the case where traceability is required, you've got a different situation altogether. As you described, you're going to have to keep a "paper trail" of all the changes you make during development. I think this activity would have to be added to the "normal" development process, and would certainly add significant overhead.

Please do send the links out in a reply to this message; I bet there are lots of folks on the list who would like to see them.
[ August 04, 2005: Message edited by: Will Gwaltney ]

[Jeanne Boyarsky]

"Srinivas",
We're pleased to have you here with us in the Process forum, but there are a few rules that need to be followed, and one is that proper names are required. Please take a look at the JavaRanch Naming Policy and adjust your display name to match it.

In particular, your display name must be a first and a last name separated by a space character, and must not be obviously fictitious.

Note that users with invalid display names are not eligible to win a book.

Thanks,
Jeanne
Forum Bartender

[Scott Ambler]

You need to be careful about situations where traceability is "required". I've seen two basic situations:
1. It's required by a paper pusher somewhere because they read the CMM(I) and thought it would be a good idea without thinking through the costs and benefits of doing so. The costs will almost always outweigh the benefits in my experience.
2. It's required by some form of audit agency. For example, in the US the Food and Drug Administration (FDA) has guidelines for IT in health-related industries which requires traceability matrices. Whether this is actually a useful thing is another discussion completely, but if your organization doesn't conform then you have a serious problem if the FDA audits your project. In these cases the cost of not conforming is often so high that you want to maintain the traces. Having said that, the FDA doesn't define how to do the traceability so you could keep it fairly simple if you wanted, although most organizations choose to go overboard.

I'd be interested to hear what people had to say about traceability tools such as Caliber RM and Requisite Pro. Are they actually adding value to your projects or are they slowing you down?

- Scott

[Ilja Preuss]

Originally posted by Shrin Krishnan:
Traceability in the sense I am using it has to do with maintaining a coherent flow from artifact to artifact as we move from thru various phases of design analysis->design->code etc. Hence, if we ever chose one use case, we can go from analysis to design easily, and in some case design to analysis.

Early analysis in Agile projects typically is very rough. XP, for example, uses so called User Stories, which at the beginning are not much more than some notes on an index card, accompanied by some informal discussion. The assumption is that we just need a very rough idea of what needs to be done to come up with a rough Release Plan, but that what gets discussed is far from being the definitive requirements. Naturally, tracing those down to parts of the final system would be of very questionable value.

It gets more interesting when we actually decide to tackle a specific feature (Story) - typically at the start of an iteration. Here the idea is to come up with a very detailed and formal requirements specification: an automated Acceptance Tests, specified (and possibly even written) by the customer.

Those automated tests provide a very usefull form of traceability: they tell us immediately when the system stops supporting the feature, by failing. And if we are interested in what some code is needed for, we can remove it and watch which tests fail, and therefore which features it supports.

Is that the kind of traceability you were speaking about?

So, analysis patterns map to design patterns

I'm rather suspicious of this connection. In my opinion, it isn't necessary, or even always good, to have a direct mapping from the problem space structure to the solution space structure.

What is your opinion on this, in your experience how is design kept coherent with change and iterations.

The source code *is* the definitive reflection of the design. Modern tools allow to explore the code very effectively. The high level view, on the other hand, needs to be in the heads of the developers, which isn't too hard, either. Some high level diagrams can help in discussions, but they aren't too hard to produce "just in time", in my experience. (And that's on a 0.5 MLOC Java project.)

[Srinivas Krishnan]

quote:
"I'm rather suspicious of this connection. In my opinion, it isn't necessary, or even always good, to have a direct mapping from the problem space structure to the solution space structure."


I respectfully disagree, I think there should be disconnect between any of the phases of the process. The simple reason being when you go through every phase you simply add layers of detail. Analysis patterns help you come up with a decent problem analysis, but there are analysis patterns that map to a design pattern.

The details are lot more than I can put in this space, let me put up a link to a paper we wrote last year, the work is a little preliminary. Hope that would convince you.

http://www.cs.unc.edu/~krishnan/publications/Patterns_Hierarchy.pdf

[Srinivas Krishnan]

correction for last post:

"....I think there should be disconnect between " I meant "...there should be no disconnect..."

[Ilja Preuss]

Well, in my experience, solving a problem (that is, going from the problem to the solution space) is a highly creative activity. If the structure of the solution were even near to obvious from looking at the problem, it wouldn't be half the fun...

But perhaps that's only true for the problems I currently have to work on.

[Dinaker Yanamandala]

The fundamental reason to have traceability is to have some sort of accountability. For example.
If you have a requirement in your requirement doc you need that to be traced to some meeting notes. This pratice helps to maintain better requirements and also gives an accountability of when and how that requirement was captured.

same applies to design. traceability to requirements from design will give you an accountability of why you designed the way you designed it with respect to a reqm

same applies to testing - makes sure each and every requirement and design is tested

Meeting notes -> Requirements -> High Lvl design -> Detail Design -> Intergation T -> System T -> UAT


DoD, FDA treat traceability matrix or docs as required artifacts

[Ilja Preuss]

Originally posted by Dinaker Yanamandala:
If you have a requirement in your requirement doc you need that to be traced to some meeting notes. This pratice helps to maintain better requirements

How does it do that?

and also gives an accountability of when and how that requirement was captured.

What do you need that for?

same applies to design. traceability to requirements from design will give you an accountability of why you designed the way you designed it with respect to a reqm

I can't imagine how this would work. Can you give an example of how this worked for you in the past? Thanks!

same applies to testing - makes sure each and every requirement and design is tested

Oh, ok. So if we had a test suite that was named "user story 1 - add customer to system", would that qualify as traceability?

Meeting notes -> Requirements -> High Lvl design -> Detail Design -> Intergation T -> System T -> UAT

On an XP project, that would probably look more like

User Story -> Customer Acceptance Tests -> Working System

How does that sound to you?

[Rick O'Shay]

Originally posted by Shrin Krishnan:
This is a question for the authors, how do you think agile methodologies handle traceability from 1 phase to another, with multiple iterations?

In my experience traceability place a key factor in developing a top notch design that can survive change. Can you provide any insight on this aspect ?

Traceability is of course a fundamental concept of software engineering, as opposed to computer programming. In the Waterfall Era we would start with a requirement and move through requirements, design, implementation and final documentation making sure that you could follow a feature from end-to-end. If you could not trace it there was a problem. This was entirely consistent with the waterfall method: a dismal unqualified failure from start to end. Traceability went out the window in week two alone with the original requirements. HAH!

[Srinivas Krishnan]

To Rick O'Shay:

So, what do you think has changed with iterative processes, that are do waterfall multiple times.


I think we require some mechanism to hold all the pieces together, without traceability people can do each phase completely seperate with no understanding of what the requirements are. And once it comes to testing and we want to see what broke where, and if it was a design flaw, we can use good traceability to localize faults and the ripple effects.

[Stan James]

I think traceability was a response to a problem with requirements that are expressed in some meeting or document and don't make it into the code. Or things that show up in the application that nobody ever really wanted. In monstrously huge and critical systems like building the shuttle it's important to make sure every engineering change is done and was really desired.

If you run a software project like that, you're probably not one of the agile fans hanging out here. My favorite Agile approach to the first problem (requirements never implemented) is writing acceptance tests up front. The second is a little tougher. We once had cowboy consultants make up amazing features in our system because they had nothing better to do than code all night in a strange town. We just told them to knock it off.

Our QA folks still maintain traceability docs from requirements to tests, I think. We're trying hard to get the tests up front so that's not necessary. I hope one day they tire of the duplication between requirements docs and tests and give up the requirements docs.

The Rose traceability matrix is definitel on the "slow you down" side of helping or hurting. We gave it up in about 1997.

[Rick O'Shay]

Just adapt to to your spiral/agile process. You have your latest feature (next iteration) based on a requirement and you can trace it through to all of your design artifacts and deliverables (a few UML diagrams and your unit tests, integration tests and functional tests). It's sort of a priori but there's nothing stopping you from incrementally creating a compendium as you go.

[Srinivas Krishnan]

Stan,

I am not suggesting using a traceability matrix like Rose. Instead maintaining traceability through patterns from artifact to artifact. The link I provided in the previous post gives an example of how it can be done.

[dinaker prasad]

Originally posted by Ilja Preuss:


On an XP project, that would probably look more like

User Story -> Customer Acceptance Tests -> Working System

How does that sound to you?

This might work for you and your organization which does not mean that it would work for everyone.

If you working within a team of 20 people and one single domain problem & architecture style traceability might not provide you with any benefits. But
When you have team size of more than 60 working on about 100 requirements(capability reqms), some of these artifacts do help.
Traceability from meeting notes to win-win conditions to requirements help people to trace the source of requirement. Aso requirement can be linked to other requirements.when a requirements is changed it gives the requirement manager to looks for reqms that can be affected because of the change. for a requirements manager he needs to account for each and every requirement he/she comes out with. For a team of 20 you might not probably have a requirements manager.
Again if you are 100% agile, with small teams working in cubes next to each other this artifact might be worthless.

And to stan remarks on tools, a simple excel sheet can do wonders. what we need to understand is when is traceability required and the extend of traceability needed.

You can only understand the beauty of some of the software artifacts and processes only when you get to see it work.
Not every one gets a chance to work with various team sizes and complexity. so when you see an instance of XP working for
you again and again that does not mean its the best solution out there for every organization.

[Ilja Preuss]

Originally posted by Srinivas Krishnan:
I think we require some mechanism to hold all the pieces together, without traceability people can do each phase completely seperate with no understanding of what the requirements are. And once it comes to testing and we want to see what broke where, and if it was a design flaw, we can use good traceability to localize faults and the ripple effects.

The solution of Agile processes to *this* problem is the following:

- don't defer testing until the end. Write the tests in parallel to implementing the features.

- fully automate all the tests. Make running them a matter of pressing a button, and observing wether they all passed a matter of watching an indicator that either turns red or green.

- run the tests very frequently, at least every night. Automate this, so that it doesn't get forgotten.

Imagine you have this process installed, and now you make a change to the design to support a new feature. Not later than the morning of the next day you will know whether your change was inappropriate for prior requirements. You will in fact know exactly which features you broke. And you will probably remember what feature you did the change for.

Would that work for you?

[Scott Ambler]

DoD, FDA treat traceability matrix or docs as required artifacts

The US DoD in practice has one of the worst, if not the worst, track record at software development in the world. Organizations which need to comply to FDA regulations, at least in my experience, are often very ineffective at software development. I'm not so sure that the rest of us should be looking to them as shiny examples of how to go about development.

Traceability is of course a fundamental concept of software engineering, as opposed to computer programming. In the Waterfall Era we would start with a requirement and move through requirements, design, implementation and final documentation making sure that you could follow a feature from end-to-end. If you could not trace it there was a problem. This was entirely consistent with the waterfall method: a dismal unqualified failure from start to end. Traceability went out the window in week two alone with the original requirements. HAH!

Yes, this is a fundamental problem with complex traceability. As long as you have someone else responsible for keeping track of the traces they get out of sync with what is actually going on. To keep them in (mostly) sync you have to slow development down to a crawl.

Unless your process has built-in traceability which is maintained by the developers you're effectively fooling yourself. The only other way I've seen it work, in practice, is to put the traceability matrix together at the end of the project. That's mostly a wasted effort to satisfy some paper pusher somewhere.

Whenever someone pushes you to support traceability, ask them when the last time was that they wrote some real code. It's incredibly rare to see someone who can actually build software think that this stuff is a good idea.

On an XP project, that would probably look more like

User Story -> Customer Acceptance Tests -> Working System

Yes, and interestingly many XPers use products like FITNesse, or just plain Wikis, which automatically include a lot of the major traces via HTML links. Use the Simplest Tools

I think traceability was a response to a problem with requirements that are expressed in some meeting or document and don't make it into the code. Or things that show up in the application that nobody ever really wanted. In monstrously huge and critical systems like building the shuttle it's important to make sure every engineering change is done and was really desired.

Exactly. And if you're building the shuttle, it might make sense. The vast majority of us aren't building the shuttle, so it probably doesn't make sense. Use the right process for the job.

Our QA folks still maintain traceability docs from requirements to tests, I think. We're trying hard to get the tests up front so that's not necessary. I hope one day they tire of the duplication between requirements docs and tests and give up the requirements docs.

Yes, traceability matrices often prove to be a band-aid on top of this problem. I would rather address the actual problem head on and instead single source information. The unfortunate problem with this approach is that the paper pushers among us don't get to justify their existence.

Just adapt to to your spiral/agile process. You have your latest feature (next iteration) based on a requirement and you can trace it through to all of your design artifacts and deliverables (a few UML diagrams and your unit tests, integration tests and functional tests). It's sort of a priori but there's nothing stopping you from incrementally creating a compendium as you go.

Yes, with an agile approach there's nothing stopping you from maintaining a trace matrix. Just understand the costs of doing so, make those costs explicit to your stakeholders, and let them decide if they wish to invest in traceability. They rarely seem to want to do so, in my experience.

Instead maintaining traceability through patterns from artifact to artifact.

Sounds like a great idea in theory, but in practice it doesn't really work for several reasons:
1. Everything isn't a pattern. The patterns community found this out the hard way a long time ago.
2. Not everyone knows all of these patterns, even if you could somehow capture everything as patterns.
3. Given an analysis pattern there are often many ways to implement it with design patterns, so the traceability isn't automatically obvious.
4. Most agilists apply patterns gently, so it wouldn't work out too well for them I suspect.

Traceability from meeting notes to win-win conditions to requirements help people to trace the source of requirement. Aso requirement can be linked to other requirements.when a requirements is changed it gives the requirement manager to looks for reqms that can be affected because of the change. for a requirements manager he needs to account for each and every requirement he/she comes out with. For a team of 20 you might not probably have a requirements manager.

Yes, for larger teams a trace matrix might add some value. But, is the team large because of the nature of the problem or because someone thought they needed a large team? Perhaps you can cut your team of 100 down to 20 simply by stripping away the bureaucrats? I've actually seen this in practice several times. In one instance a team of 80 people where doing the work of a team of 10 people (the organization was CMMI level 3) but they didn't know any better. The Standish Group once ran into two virtually identical projects (from a requirements point of view). One org decided it was a small project and delivered with a team of 5-7 in less than a year. The other org looked at the same initial requirements, decided it was a large project, and failed to deliver after 5 years with a team which ranged from 80-120 people.

Imagine you have this process installed, and now you make a change to the design to support a new feature. Not later than the morning of the next day you will know whether your change was inappropriate for prior requirements. You will in fact know exactly which features you broke. And you will probably remember what feature you did the change for.

Yes. There are many ways to achieve these goals. Agilists have found quick, effective, and inexpensive ways to do so. Traditionalists seem to cling to documentation-heavy ways which in practice don't seem to work very well. Unfortunately, as I pointed out earlier, with agile techniques the paper pushers find it very difficult to find a role for themselves in agile techniques. It makes you wonder if the existing processes which many orgs currently follow reflect the actual needs of the orgs, or the needs of the paper pushers to justify their existence.

- Scott

[Ilja Preuss]

Originally posted by dinaker prasad:
This might work for you and your organization which does not mean that it would work for everyone.

Well, yes, of course not. It was just an example of how I see vanilla XP work, as an example of an Agile process. Now what I would be interested in is discussing what more people might need, and why - and how they could get it in an as Agile way as possible (as this was the question of the original poster, the way I understood it).

Traceability from meeting notes to win-win conditions to requirements help people to trace the source of requirement.

I can certainly imagine putting the meeting date and perhaps the customer name on a user story card, if I feel that that would help, yes.

Aso requirement can be linked to other requirements.when a requirements is changed it gives the requirement manager to looks for reqms that can be affected because of the change. for a requirements manager he needs to account for each and every requirement he/she comes out with. For a team of 20 you might not probably have a requirements manager.

You are right, I've never worked together with a requirements manager. It sounds like he would take the role of the XP customer, though. He certainly should do what he needs to do to make sure that the system becomes coherent. Some people, for example, assign user stories to "themes".

Again if you are 100% agile, with small teams working in cubes next to each other this artifact might be worthless.

As an aside, if you want to be even near to "100% agile" (whatever that would mean), you probably should do away with cubes as one of the first steps...

You can only understand the beauty of some of the software artifacts and processes only when you get to see it work.
Not every one gets a chance to work with various team sizes and complexity. so when you see an instance of XP working for
you again and again that does not mean its the best solution out there for every organization.

Yes, it's good to remember that. And it doesn't only apply to XP, of course.