Why Agile Works: Feedback

[Scott Ambler]

I just posted Why Agile Software Development Techniques Work: Improved Feedback and was hoping to get some feedback.

Thanks in advance.

- Scott

[Ilja Preuss]

In my opinion, most confusion about the "cost of change curve" stems from the fact that there are in fact two different curves, but often enough people don't distinguish between them.

The first, the one you discuss in the article, and I think the one Boehm referred to, is the cost of changing the code to repair a bug. That curve still isn't flat, and the short feedback cycles work as you describe. (Although I think even Boehm recently said that with modern techniques the curve can be much more flat than he originally stated.)

The second curve is the one of the cost of introducing a new feature into the code base. I'd argue that in Agile projects this curve actually *needs* to be flat, almost by definition. And the techniques to keep it flat are fully automated testing and merciless refactoring.

[Scott Ambler]

Actually, at Agile Cost of Change Curve I argue that the flat curve is really just the left-hand side of the traditional exponential curve.

- Scott

[Stan James]

Another significant factor in cost of change is the recent improvement in languages and tools. This week I focused on package dependencies in my Wiki and extracted an interface for a key class in seconds, dragged the interface to another package, ran the test suite and went on my way. Making changes so cheaply enables YAGNI design. It is much less necessary to design all the good ideas up front.

Back in my COBOL days a single change to a copybook could eat up days of impact analysis, code changes, manual builds and testing. Don't miss that a bit!

[Gerardo Tasistro]

Ever tried proving this in a mathematical model? Aka control system. With feedback channels. There should be more than enough data to map and compare systems. Compare an no feedback-little feedback system:



With ones with feedback:





Of course given the discrete nature of a programming environment. Your response might look stepped.



At a short glance this issue compares to the spring, damper and mass problem. The bigger the problem (requirement list-mass) and the bigger the "bureocracy" (prestartup time, design, BRUF) the longer it will take you to get it moving. And once you spend an enormous amout of energy (read money) getting it moving you will undoubtly overshoot (because your feedback system aka management is pressing hard). You overshoot with this big mass (but also take a long time to notice you overshot, because of lack of proper feedback - read fast feedback). And now your spending a lot of energy (read money) in the opposite direction to fix the overshoot. Which will undershoot (due to feedback issues) and on and on it will go.

With proper backing data I'm sure this can be modeled.


Images from http://www.engin.umich.edu/group/ctm/PID/PID.html and
http://www.engin.umich.edu/group/ctm/digital/digital.html

[Ilja Preuss]

Originally posted by Scott Ambler:
Actually, at Agile Cost of Change Curve I argue that the flat curve is really just the left-hand side of the traditional exponential curve.

Yes, I understand, and kind of agree - for the first type of curve.

In fact, if I remember correctly, the original curve even put *phases* at the x axis - finding a bug in testing was more costly than finding it in coding, than finding it in design etc.

So on the other hand it could be argued that for an approach that doesn't use phases, the curve simply doesn't apply. Using TDD, finding a bug in testing instead of coding simply doesn't make sense...

Or you could say that Agile teams compress the timeline, and therefore also compress the costs. Finding a problem during the TDD cycle still will be exponentially more costly than finding it during the initial discussion with the customer, but we are talking about days here, not months, so the effect is strongly curbed (is that the right word?).

The longer I think about this, the more I get the feeling that the curve is that much of an oversimplification that it could support any argument, if just interpreted "correctly"...

Anyway, talking about the other curve, early feedback alone won't help you keeping the cost of adding features late in the project small - if you allow your code/design to deteriorate over time, no amount of feedback will keep the costs of adding new features even nearly constant.

[Ilja Preuss]

Originally posted by Stan James:
Another significant factor in cost of change is the recent improvement in languages and tools.

Very true. Your example resonates well with my "merciless refactoring" comment, I think. The more costly refactoring, the less Agile development has to be, or so it seems to me.

And don't forget pure computing power - it enables a lot of the feedback Scott speaks about: fast compilers and test suites etc. I'm waiting for the day that a tool like Jester - showing which lines of production code don't contribute to a test case suceeding - can be run in the background while you type, just like syntax checking in modern IDEs does today.

Another important part is reducing the overhead of bureaucracy. There is much less value in finding a problem days instead of months after a feature got discussed with the customer - if once you've identified the problem, you need to spend days to write documents and get them signed (for example) before you can put the new knowledge into code.

[Ilja Preuss]

Originally posted by Gerardo Tasistro:
Ever tried proving this in a mathematical model?

I think such a model can provide interesting insights to someone who's mind already is open to the idea of Agile development.

Using such a model to convince a skeptic sounds rather inept to me - after all, you can find a model for any kind of curve, and coming up with the "right" model is highly subjective. As a consequence, such a model implies a connotation of "objective science" that is misleading. A skeptic will likely notice that and feel more manipulated than enlightened...

[Scott Ambler]

From Stan James:
Another significant factor in cost of change is the recent improvement in languages and tools. This week I focused on package dependencies in my Wiki and extracted an interface for a key class in seconds, dragged the interface to another package, ran the test suite and went on my way. Making changes so cheaply enables YAGNI design. It is much less necessary to design all the good ideas up front.

Back in my COBOL days a single change to a copybook could eat up days of impact analysis, code changes, manual builds and testing. Don't miss that a bit!

Exactly. This is one of the reasons why we've seen such a large number of new OSS tools coming from agile practitioners. Without the tools, you can't move quickly.

From Gerardo Tasistro
Ever tried proving this in a mathematical model?

No, but it is something I would expect eventually from the research community.

From Ilja Preuss:
In fact, if I remember correctly, the original curve even put *phases* at the x axis - finding a bug in testing was more costly than finding it in coding, than finding it in design etc.

So on the other hand it could be argued that for an approach that doesn't use phases, the curve simply doesn't apply. Using TDD, finding a bug in testing instead of coding simply doesn't make sense...

Yes, the original curve had phases along the bottom because that was the way that people thought back then, and still do for the most part. When you read the material behind the curve, however, the discussion quickly became one of time, not phases.

From Ilja Preuss:
Anyway, talking about the other curve, early feedback alone won't help you keeping the cost of adding features late in the project small - if you allow your code/design to deteriorate over time, no amount of feedback will keep the costs of adding new features even nearly constant.

Yes, that's why I start the article saying that reducing the feedback cycle is one of several reasons why agile works. There will be future articles in this series. ;-)

From Ilja Preuss:
Another important part is reducing the overhead of bureaucracy. There is much less value in finding a problem days instead of months after a feature got discussed with the customer - if once you've identified the problem, you need to spend days to write documents and get them signed (for example) before you can put the new knowledge into code.

Yes, this is why agile practices such as XP's Onsite Customer or better yet Active Stakeholder Participation are so critical to your success.

- Scott

[Michael Duffy]

Originally posted by Gerardo Tasistro:
Ever tried proving this in a mathematical model? Aka control system. With feedback channels. There should be more than enough data to map and compare systems. Compare an no feedback-little feedback system:



With ones with feedback:





Of course given the discrete nature of a programming environment. Your response might look stepped.



At a short glance this issue compares to the spring, damper and mass problem. The bigger the problem (requirement list-mass) and the bigger the "bureocracy" (prestartup time, design, BRUF) the longer it will take you to get it moving. And once you spend an enormous amout of energy (read money) getting it moving you will undoubtly overshoot (because your feedback system aka management is pressing hard). You overshoot with this big mass (but also take a long time to notice you overshot, because of lack of proper feedback - read fast feedback). And now your spending a lot of energy (read money) in the opposite direction to fix the overshoot. Which will undershoot (due to feedback issues) and on and on it will go.

With proper backing data I'm sure this can be modeled.


Images from http://www.engin.umich.edu/group/ctm/PID/PID.html and
http://www.engin.umich.edu/group/ctm/digital/digital.html

I'm not sure how far this line of thought will go, but I absolutely love the approach. It warms the cockles of my engineering heart.

In my opinion the problem of agile development seems to fall into the social sciences. It'll suffer from the same problems as applying physics to other non-physical systems.

I work at a large company where waterfall is encased in concrete. I love talking about these things, but I fear that I'll never be able to change the organization. "If you can't change your organization, change your organization."

Just two comments that I'd like Scott et. al. to address:

(1) Maybe BUFD is verboten, but does that necessarily mean NO DESIGN up front? That feature has always smacked of development without adult supervision to me.

(2) What does one do about team members who don't go along? Agility always seems to require a team with a common mindset and ability. I think it breaks down when there's too wide a gap in abilities.

(3) What does agility do about "oil canning" of code? Programmer A writes something and checks it in. Programmer B sees it, decides to refactor since all code is held in common, and checks it in. Ad nauseum. (I've heard a story from a colleague where this actually happened.)

An interesting take. I'm just not sure that it's not too idealized. Does the curve only hold for "pure" agile practice? How do the forces of project managers, etc. change the shape of the curve?

[Gerardo Tasistro]

Originally posted by Michael Duffy:


(1) Maybe BUFD is verboten, but does that necessarily mean NO DESIGN up front? That feature has always smacked of development without adult supervision to me.

I don't believe in a "NO DESIGN up front" model. I do believe strongly in feedback and constant evaluation of your project.

If asked to define Agile or XP I'd say: "A set of rules and recomendations for the integration and usage of tools and techniques to achieve non-linear software development."

With this I cover the fact that you're still using the same stuff. You don't need to change language or IDE. Your "fixing" the way you work with them in acordance to a set of "good practices" that lead to a non-linear software development. With non-linear I mean to say one which is not deterministic (like watefall for example). One which might seem chaotic at times, but strangely for some (the skeptics) does deliver.

I've been cooking some ideas in my head. I think you can model this well with forces taking effect. I'll work up some graphs to try and explain this, but for the moment being I have to get some breakfast. Tacos are waiting and that barbacoa just smell soo good

Catch y'all in a bit.

[Michael Duffy]

Originally posted by Gerardo Tasistro:


I don't believe in a "NO DESIGN up front" model. I do believe strongly in feedback and constant evaluation of your project.

I agree - I believe that design has to have a place.

If asked to define Agile or XP I'd say: "A set of rules and recomendations for the integration and usage of tools and techniques to achieve non-linear software development."

I think you leave most people behind when you throw in the "non-linear" phrase. I happen to know what it means in a physics context, but I'm not sure that the definition would be so precise in software. If I interpret you correctly, you're calling waterfall linear because design is followed by implementation is followed by test. You're saying agile is non-linear because it's iterative, a la Newton-Raphson iteration. Correct?

With this I cover the fact that you're still using the same stuff. You don't need to change language or IDE. Your "fixing" the way you work with them in acordance to a set of "good practices" that lead to a non-linear software development. With non-linear I mean to say one which is not deterministic (like watefall for example). One which might seem chaotic at times, but strangely for some (the skeptics) does deliver.

If only waterfall was deterministic. 8) I disagree with that notion.

There's the larger question of team size. I thought agility has been demonstrated with smaller teams ("the lunch table criterion") physically located in the same place, preferrably co-resident with their business partners. What about larger teams with off-shore components?

I've been cooking some ideas in my head. I think you can model this well with forces taking effect. I'll work up some graphs to try and explain this, but for the moment being I have to get some breakfast. Tacos are waiting and that barbacoa just smell soo good

Catch y'all in a bit.

It's snowing again here. A nor'easter has brought winter back to New England.

[Gerardo Tasistro]

Well my usage of the term deterministic is NOT in the observed behavior of the waterfall model, but rather in the expected. Waterfall advocates WANT it to be deterministic. That is a "with no randomness well cut out clear objective". That is why it is cut into phases and all this time is spent in the design phase. In the false belief that it will lead to a succesful estimation of needs, costs and time. "We have it all covered, what could go wrong?" Well just about everything.

Non-linear on the other hand expects all sorts of things to go "wrong". It understands that the system is very very sensitive to initial conditions. I'm using the term non-linear in the context of a very complex system (not easily modeled by linear equations). This non-linear (read complex system) requires a lot of quick feedback to keep it on track. But more important that the understanding of the need for feedback is the understanding of the limitations of our predictions. Given the sensitivity to initial conditions we must comprehend that we can NOT see very far into the future and be correct with our estimations.

Now what I find fascinating about this conversation is how much future is "future". And that when you sit down to model this agile and waterfall become the same thing. Except one has different coefficients than the other (and do consider that those coefficients can take the value of cero (0)). A system with 0 on all its feedback channels is effectively a system without feedback.

It is clear that agile doesn't want to look too far into the future while waterfall does. That looks alot like the z^(n) and the whole sample and holding issue with discrete control systems. It is an immediate conclusion that if you don't sample you can't control. So agile already has a head start there and that should ring a bell (read warning) to the waterfall advocates. Agile also samples a lot faster. They're always checking. So no matter how fast the project changes (which we can equate to base/natural frequency of a signal) agile is bound to win since they sample more often and are bound to have a good enough sampling for the "frequency" of the project. This should sound as another warning call for waterfall advocates, but it is usually dismissed as "chaos" or "disorder".

They see agile's short term goals and lack of long term planning as disorder. Without realizing that their own plans have very low long term value given the non-linear nature of the job at hand. Agile's short term planning is a signal of short sampling times (or in other words high sampling frequency). Your systems output control signal can't last longer at a set value than the sample period. Because obviously it will be altered by such sample. In other words next monday's plan will be altered by friday's evaluation. So making a perfectly crisp 2 or 3 week plan last monday was useless.

Now I also used the term "pefectly crisp" to clarify something. That is you can have a look ahead plan for the next month and a todo list for this week. That can also be modeled through filters and more sample and hold or further holding. Your current feedback is not only altered by the previous sample, but by the sample prior to that and the one prior and prior etc. Up to a limit of course.

It would be really interesting to try and plot this in a Zplane or something similar. Make a model of all these feedbacks. Get a system and plot the ceros and poles. That would be key to finding the stability of the system. Got to get back to my digital control books.

Anyway to sum it up there is more than enough evidence to prove agile wins over waterfall. The sampling alone element should sufice. More arguments can be built on a model.

For example why can't the project reach its goals (overdamped system)? Why are we bouncing around our objective and not reaching it (subdamped system)? The first could be a management that ties too much resources to planning so we never quite get there. Too much energy is spent seeing were the target is there is no left to actually go there. The second case could be a case of too little management creating the inability to commit to a goal. Lets go this way, then the other way then this way again. Your object oscillates around "very good ideas".

IMHO life is not linear and project management shouldn't be either.

[Michael Duffy]

Well my usage of the term deterministic is NOT in the observed behavior of the waterfall model, but rather in the expected. Waterfall advocates WANT it to be deterministic. That is a "with no randomness well cut out clear objective". That is why it is cut into phases and all this time is spent in the design phase. In the false belief that it will lead to a succesful estimation of needs, costs and time. "We have it all covered, what could go wrong?" Well just about everything.

I agree with your point about "no randomness".

Non-linear on the other hand expects all sorts of things to go "wrong". It understands that the system is very very sensitive to initial conditions. I'm using the term non-linear in the context of a very complex system (not easily modeled by linear equations). This non-linear (read complex system) requires a lot of quick feedback to keep it on track. But more important that the understanding of the need for feedback is the understanding of the limitations of our predictions. Given the sensitivity to initial conditions we must comprehend that we can NOT see very far into the future and be correct with our estimations.

Again, well said.

Now what I find fascinating about this conversation is how much future is "future". And that when you sit down to model this agile and waterfall become the same thing. Except one has different coefficients than the other (and do consider that those coefficients can take the value of cero (0)). A system with 0 on all its feedback channels is effectively a system without feedback.

It is clear that agile doesn't want to look too far into the future while waterfall does. That looks alot like the z^(n) and the whole sample and holding issue with discrete control systems. It is an immediate conclusion that if you don't sample you can't control. So agile already has a head start there and that should ring a bell (read warning) to the waterfall advocates. Agile also samples a lot faster. They're always checking. So no matter how fast the project changes (which we can equate to base/natural frequency of a signal) agile is bound to win since they sample more often and are bound to have a good enough sampling for the "frequency" of the project. This should sound as another warning call for waterfall advocates, but it is usually dismissed as "chaos" or "disorder".

In my current situation project managers see themselves as the governors and controllers of all non-linear behavior. Think of them as big dampers. (I'm not arguing that this is good or desireable, just what is.)

I find PMs making business and technical decisions, either explicitly ("That's not part of scope") or implicitly by filtering what's presented to business and technical people. Very harmful.

They see agile's short term goals and lack of long term planning as disorder. Without realizing that their own plans have very low long term value given the non-linear nature of the job at hand. Agile's short term planning is a signal of short sampling times (or in other words high sampling frequency). Your systems output control signal can't last longer at a set value than the sample period. Because obviously it will be altered by such sample. In other words next monday's plan will be altered by friday's evaluation. So making a perfectly crisp 2 or 3 week plan last monday was useless.

Now I also used the term "pefectly crisp" to clarify something. That is you can have a look ahead plan for the next month and a todo list for this week. That can also be modeled through filters and more sample and hold or further holding. Your current feedback is not only altered by the previous sample, but by the sample prior to that and the one prior and prior etc. Up to a limit of course.

It would be really interesting to try and plot this in a Zplane or something similar. Make a model of all these feedbacks. Get a system and plot the ceros and poles. That would be key to finding the stability of the system. Got to get back to my digital control books.

Anyway to sum it up there is more than enough evidence to prove agile wins over waterfall. The sampling alone element should sufice. More arguments can be built on a model.

If that were true, agile would be far more widespread. My experience that it's not either says that I need to change my situation or your data needs updating. It's not widespread from my vantage point.

For example why can't the project reach its goals (overdamped system)? Why are we bouncing around our objective and not reaching it (subdamped system)? The first could be a management that ties too much resources to planning so we never quite get there. Too much energy is spent seeing were the target is there is no left to actually go there. The second case could be a case of too little management creating the inability to commit to a goal. Lets go this way, then the other way then this way again. Your object oscillates around "very good ideas".

IMHO life is not linear and project management shouldn't be either.[/QB]

Agreed. Let's keep talking.

[Ilja Preuss]

Originally posted by Michael Duffy:
If that were true, agile would be far more widespread.

In my experience, resistance to Agility has much more to do with culture and not wanting to leave ones comfort zone, than with rational reasons.

[Ilja Preuss]

Originally posted by Gerardo Tasistro:

It is clear that agile doesn't want to look too far into the future while waterfall does.

Not much to disagree with, so let me nitpick on this one:

I'd rather say that *looking* into the future is ok - or in fact having some bright vision of the far future even encouraged.

What Agilists shun is *committing* to plans about all but the nearest future. Instead they prefer to keep their options open as long as possible (Lean Software Development calls this "set based decision making").

(I notice that the latter is probably what you meant, but I've seen so many people misunderstanding this to mean that you are disallowed to even think a little bit about the future, that I tend to be a little bit anal about this.)
[ February 12, 2006: Message edited by: Ilja Preuss ]

[Michael Duffy]

Originally posted by Ilja Preuss:


In my experience, resistance to Agility has much more to do with culture and not wanting to leave ones comfort zone, than with rational reasons.

Certainly emotional, but it might still be rational. I don't think that agile success is always so cut & dried.

I can see where a team might have tried to adopt it in the early enthusiasm, have a project fail for any number of reasons, and have that failure become part of cultural dogma: "We tried that once. Remember what a mess project X was? We won't make that mistake again."

I think that agility does require some enlightened leadership. Agility implies a democracy of equals, but I'd argue that the level of even the worst developer on those teams is relatively high. It's not the right place for coders who want nothing more than to put their heads down and be told what to do all day long. Agility implies a certain passion and introspection, a desire to always be better. Unfortunatly, all those qualities vary where I am.

[Michael Duffy]

Originally posted by Michael Duffy:


Certainly emotional, but it might still be rational. I don't think that agile success is always so cut & dried.

I can see where a team might have tried to adopt it in the early enthusiasm, have a project fail for any number of reasons, and have that failure become part of cultural dogma: "We tried that once. Remember what a mess project X was? We won't make that mistake again."

I think that agility does require some enlightened leadership. Agility implies a democracy of equals, but I'd argue that the level of even the worst developer on those teams is relatively high. It's not the right place for coders who want nothing more than to put their heads down and be told what to do all day long. Agility implies a certain passion and introspection, a desire to always be better. Unfortunatly, all those qualities vary where I am.

Your agile team is in good shape if you've got Martin Fowler or Scott Ambler or Kent Beck leading it. But what about all those places where strong leadership like that isn't available? When you have a democracy of equals, who breaks the ties? I'd argue that a person of stature can be the final arbiter, but that's not always so universally agreed upon.

[Ilja Preuss]

Originally posted by Michael Duffy:
I think that agility does require some enlightened leadership.

Above all, it requires that the leaders grok the values of Agility, instead of just trying to apply some practices.

Agility implies a democracy of equals

This doesn't sound right to me, but I'm not sure I understand what you are getting at. Care to elaborate?

It's not the right place for coders who want nothing more than to put their heads down and be told what to do all day long. Agility implies a certain passion and introspection, a desire to always be better. Unfortunatly, all those qualities vary where I am.

I think that Agility also *induces* passion for the work - or rather that most developers would have it anyway, if it weren't actively suppressed by the working environment.

Of course that doesn't mean that every developer will instantly become passionate about working in an Agile team. It needs a very sensitive coach to lead such a team to more self organization and responsibility.

[Ilja Preuss]

Originally posted by Michael Duffy:
Your agile team is in good shape if you've got Martin Fowler or Scott Ambler or Kent Beck leading it. But what about all those places where strong leadership like that isn't available?

It works without Fowler, Ambler or Beck, really. (Sorry Scott, but that's the way it is...)

When you have a democracy of equals, who breaks the ties?

Well, I wouldn't call a team a democracy. Anyway...

If a team has trouble deciding on something, there are a number of solution, depending on what the actual problem is:

- continue the discussion, (see above), until everyone really feels heard and taken care of, or

- continue the discussion, if possible with the help of an experienced facilitator, to find the missing creative solution everyone can live with (I highly recommend reading "The Facilitators Guide to Participatory Decision Making"), or

- in the case of a discussion under missing data, flip a coin and gather feedback; set a date in the future where the results will be reviewed and further action be discussed, or

- in the case of someone who *really* just wants to be trouble, or who doesn't fit in the team, let the project manager make him work on a different project. (I have never seen this last option being necessary.)

[Ilja Preuss]

Originally posted by Michael Duffy:
There's the larger question of team size. I thought agility has been demonstrated with smaller teams ("the lunch table criterion") physically located in the same place, preferrably co-resident with their business partners. What about larger teams with off-shore components?

If I remember correctly, Thoughtworks has a case study online for an Agile project that was (partially?) outsourced to India.

And regarding "big" projects, Jutta Eckstein has coached some Agile projects in Germany with hundreds of developers; she has also written a book about it: http://www.jeckstein.de/agilebook/index.html

[Michael Duffy]

Originally posted by Ilja Preuss:


If I remember correctly, Thoughtworks has a case study online for an Agile project that was (partially?) outsourced to India.

And regarding "big" projects, Jutta Eckstein has coached some Agile projects in Germany with hundreds of developers; she has also written a book about it: http://www.jeckstein.de/agilebook/index.html

Sounds like some new information that I wasn't aware of. Thanks for cluing me in, Ilja. I'll have to look for the study and the book.

[Michael Duffy]

I think that Agility also *induces* passion for the work - or rather that most developers would have it anyway, if it weren't actively suppressed by the working environment.

Of course that doesn't mean that every developer will instantly become passionate about working in an Agile team. It needs a very sensitive coach to lead such a team to more self organization and responsibility.

I agree. I'm fortunate enough to work with two young, smart guys who understand the value of agility. We've been trying to infect the rest of the team, from management to architects to developers, about better practices than the waterfall methodology that's firmly ensconced.

In spite of numerous attempts to praise the virtue of test first and JUnit, has anyone else written a single unit test? No. We all use it, but others aren't following.

There's the larger question: How do you make agility more widespread?

[Ilja Preuss]

Originally posted by Michael Duffy:
I love talking about these things, but I fear that I'll never be able to change the organization. "If you can't change your organization, change your organization."

I've done the latter for some times, but today I think it was more a problem of me not being able to speak to management in terms they would have understood - and of being too hotheaded...

Seriously, if you're interested in the first part of that quote, I can highly recommend another book: "Fearless Change" contains a lot of good tips on how to get people to being interested in changing the way they work.

(1) Maybe BUFD is verboten, but does that necessarily mean NO DESIGN up front?

I don't even think NO DESIGN is *possible*. I *need* to have some idea about a rough possible design to be able to even estimate a feature.

That feature has always smacked of development without adult supervision to me.

I don't think I understand that analogy...

What does one do about team members who don't go along? Agility always seems to require a team with a common mindset and ability.

Team building is an important skill for an Agile coach. Having said that,I don't think that a common mindset or ability is even desirable within an Agile team. What you need are team members who's first priority it is to help the team reach it's common goal. If that is not the case, that is a management problem.

I think it breaks down when there's too wide a gap in abilities.

I don't think so. In fact I think that a wide gap in abilities is *less* of a problem in an Agile team.

What does agility do about "oil canning" of code? Programmer A writes something and checks it in. Programmer B sees it, decides to refactor since all code is held in common, and checks it in. Ad nauseum. (I've heard a story from a colleague where this actually happened.)

Sounds like not enough communication to me. Why didn't B tell A that he is going to refactor that code? Why didn't A tell B that he doesn't agree with his refactoring? Why didn't they simply Pair Program for a short while to resolve the issue?

[Michael Duffy]

This doesn't sound right to me, but I'm not sure I understand what you are getting at. Care to elaborate?

I'll try.

If the agile ideal is that code is held in common, and anyone can refactor at any time, there can arise a situation where two programmers A and B feel equally passionately about a particular point of interest. Let's say A believes in DAOs, and B thinks that results in an anemic domain model. Both are strong-willed individuals, both are considered capable by the team. Their debate is played out during the design sessions, without a unanimous decision from the rest of the team. Who decides? Must it be unanimous? Or will the project be compromised by hurt feelings or half-hearted support?

Agility and XP has always impled that confederation of equals to me. Everyone participates in the design, and anyone can refactor anyone else's code. It's collective ownership and responsibility, which means collective decisions and equal responsibility.

Agreed?

[Ilja Preuss]

Originally posted by Michael Duffy:
In spite of numerous attempts to praise the virtue of test first and JUnit, has anyone else written a single unit test? No. We all use it, but others aren't following.

In my experience, praise doesn't work. What works is action.

Are you ever asked to help a coworker debug some of his code? That might be a good opportunity to write some unit tests with him.

[Michael Duffy]

I don't even think NO DESIGN is *possible*. I *need* to have some idea about a rough possible design to be able to even estimate a feature.

Excellent, we agree then.

My first reading of XP made such a point of "NO BUFD" that it sounded like "no design". Iterations and off you go!

"That feature has always smacked of development without adult supervision to me." - I don't think I understand that analogy...

It might be my misreading what XP is saying, but rough division into iterations didn't seem like enough to go on to start.

Can large, enterprise systems be designed organically like this? I have no experience with it. My efforts have been smaller scale.

[Michael Duffy]

Originally posted by Ilja Preuss:


In my experience, praise doesn't work. What works is action.

Are you ever asked to help a coworker debug some of his code? That might be a good opportunity to write some unit tests with him.

Yeah, I've done it, and so have the others. We did get one guy to drink from the Kool-Aid and get religion, but that was after a year's effort (and a successful project where his beeper isn't keeping him up at night). But it's been slow going.

You're probably right - it'll take more 1:1 pairing to take root.

[Ilja Preuss]

Originally posted by Michael Duffy:
Let's say A believes in DAOs, and B thinks that results in an anemic domain model. Both are strong-willed individuals, both are considered capable by the team. Their debate is played out during the design sessions, without a unanimous decision from the rest of the team.

A facilitator would probably try to get the "silent majority" become more vocal. Not that *I*'m good at such stuff...

Who decides? Must it be unanimous? Or will the project be compromised by hurt feelings or half-hearted support?

I think the most important part is having everyone understand that the team coming to a common decision has top priority - but that that decision doesn't need to be final.

By the way, how would a non-Agile team solve that dilemma - while avoiding "half-hearted support"?

Agility and XP has always impled that confederation of equals to me. Everyone participates in the design, and anyone can refactor anyone else's code. It's collective ownership and responsibility, which means collective decisions and equal responsibility.

Agreed?

I have to think about it. Something doesn't sound right, but I currently can't put a finger on it...

[Ilja Preuss]

Originally posted by Michael Duffy:

But it's been slow going.

Yeah, it takes a *lot* of patience - that's the top one thing I had to learn...

[Ilja Preuss]

Originally posted by Michael Duffy:

Can large, enterprise systems be designed organically like this? I have no experience with it. My efforts have been smaller scale.

I actually think that's the only way they can be designed.

"Every big successful system started as a small successful system" - who said that again?

[Michael Duffy]

Originally posted by Ilja Preuss:

I actually think that's the only way they can be designed.

You must be right, because all the enterprise systems where I work are steaming piles. It's a wonder they survive in business at all. They're firmly in the waterfall camp, and PMs rule the earth.

"Every big successful system started as a small successful system" - who said that again?

Not the same person who said "Every unsuccessful system started as a dream of success". 8) I wish I could take credit, but I can't.

[Michael Duffy]

By the way, how would a non-Agile team solve that dilemma - while avoiding "half-hearted support"?

Good question, because it's not specific to agile teams.

Where I work, the technical people might be overruled by the PM or the business people (note: non-technical people making technical decisions). The hurt feelings amongst technical people remain unresolved.

Sometimes technical people can resolve it. Our last project had an argument about DTOs that was decided in favor of NOT using them. It made for one heated design session, but the rest was smooth.

[Michael Duffy]

Originally posted by Michael Duffy:


Good question, because it's not specific to agile teams.

Where I work, the technical people might be overruled by the PM or the business people (note: non-technical people making technical decisions). The hurt feelings amongst technical people remain unresolved.

Sometimes technical people can resolve it. Our last project had an argument about DTOs that was decided in favor of NOT using them. It made for one heated design session, but the rest was smooth.

It's a snowy day. If my wife doesn't have plans for me, I might stop to write some code and try to be a bit smarter. Thanks for the craic, Ilja. Most enlightening and enjoyable. Sincerely, MOD

[Ilja Preuss]

You're very welcome!

By the way, I just remembered that the Agile Alliance has a well sorted article section:

http://agilealliance.org/articles/AgileArticlesCatSearch?category=Scaling

http://agilealliance.org/articles/AgileArticlesCatSearch?category=Offshore%20Development

[Scott Ambler]

Addressing several issues at once, sorry for any confusion.

(1) Maybe BUFD is verboten, but does that necessarily mean NO DESIGN up front? That feature has always smacked of development without adult supervision to me.

We definitely model up front, see AMDD-Initial Modeling. It's just that we don't waste our time writing lots of documentation that will have to change, which might not even be read, and if it was might not be understood.

There's the larger question of team size. I thought agility has been demonstrated with smaller teams ("the lunch table criterion") physically located in the same place, preferrably co-resident with their business partners. What about larger teams with off-shore components?

My March column in Software Development overviews techniques for scaling agile software development. The print magazine is out, and the online version should be available sometime this coming week.

Agile teams get quite large. The Eclipse project, for example, follows an agile process and it's a large team that clearly isn't co-located.

If that were true, agile would be far more widespread. My experience that it's not either says that I need to change my situation or your data needs updating. It's not widespread from my vantage point.

My May column addresses the issue of whether we've crossed the chasm or not. Some techniques and methods have, some haven't. You might find Where's the Proof That Agile Works? interesting in the mean time. The short story is that where your organization is on the adoption curve is one determining factor as to whether agile is something you want to consider. The further to the right on the curve, the less chance it's attractive to you yet.

I think that agility does require some enlightened leadership. Agility implies a democracy of equals, but I'd argue that the level of even the worst developer on those teams is relatively high. It's not the right place for coders who want nothing more than to put their heads down and be told what to do all day long. Agility implies a certain passion and introspection, a desire to always be better. Unfortunatly, all those qualities vary where I am.

Right now it requires enlightened leadership because we're somewhere around the chasm.

You're right, agile developers do seem to be amongst the better folks out there. One issue is that agile requires generalizing specialists, and by their very nature those are the better people out there. Coincidently, my June column is going to address how traditional folks, typically specialists, can be useful on agile projects. It might not be what some people want to hear, but hopefully it'll be a wake up call for others.

- Scott

[Michael Duffy]

Originally posted by Scott Ambler:
Addressing several issues at once, sorry for any confusion.



Right now it requires enlightened leadership because we're somewhere around the chasm.

You're right, agile developers do seem to be amongst the better folks out there. One issue is that agile requires generalizing specialists, and by their very nature those are the better people out there. Coincidently, my June column is going to address how traditional folks, typically specialists, can be useful on agile projects. It might not be what some people want to hear, but hopefully it'll be a wake up call for others.

- Scott

Thanks, Scott. I have been reading your stuff in "SD". I'll look for those columns.

[Don Morgan]

One of the most critical success factors for software development projects is the quality of the communication on the team. Any strong, successful team realizes this and will place an emphasis on communication, regardless of the specifics of the life cycle it follows. Everyone knows face-to-face communication is the best form of communication, and most Agile practices encourage it (as opposed to generating reams and reams of documentation).

Also, the earlier you can get feedback, i.e., timely communication, the better. This is why using an iterative life cycle helps, keeping to the lower portion of the curve. Even for Agile projects, the curve is not flat. Changing requirements mean more work needs to be done, and, in addition to the actual coding changes, even if less documentation needs to be updated, the effort to maintain suites of automated test cases in the face of changing requirements should not be underestimated.

In summary then, agile works by recognizing, fostering and encouraging strong and timely communication; strong due to the nature of face-to-face communication, and timely by reducing the length of feedback cycles.
[ February 13, 2006: Message edited by: Don Morgan ]

[Ilja Preuss]

Originally posted by Michael Duffy:
Agility and XP has always impled that confederation of equals to me. Everyone participates in the design, and anyone can refactor anyone else's code. It's collective ownership and responsibility, which means collective decisions and equal responsibility.

I think the problem I had with this statement is the "anyone can refactor anyone else's code" part. Although it's basically true, there needs to be a team culture that steers when it's ok to just refactor the code, and when you need to consult the original author. It's something a team that isn't used to this way of working actively needs to learn.

Similarly with collective decisions - the team members need to learn that they sometimes will have to live with team decisions they don't fully agree with - and to decide for themselves which decisions are important enough to be picky about.

Coincidentally I'm just reading Engines of Democracy. It's a very interesting read, although it's not about software.

The average group of 15 or 16 people can't reach consensus on where to go for lunch -- let alone how to run a factory. How to organize a production line, whether to hire someone, how to assess someone's skills for promotion, even how to pick who will work over the weekend -- those kinds of issues inspire strong disagreement. "Everybody doesn't see things in the same way," says Williams. "But we've had training on how to reach consensus. We've had training on how to live with ideas that we might not necessarily agree with." And the team members always have the power to change things that don't work out. Says Williams: "All the things you normally fuss and moan about to yourself and your buddies -- well, we have a chance to do something about them. I can't say, 'They' don't know what's going on, or, 'They' made a bad decision. I am 'they.' "

The rest is very interesting, too...

[Don Morgan]

One of several reasons why agile techniques are so effective, in my opinion, is that they reduce the feedback cycle between the generation of an idea (perhaps a requirement or a design strategy) and the realization of that idea. This not only minimizes the risk of misunderstanding, it also reduces the cost of addressing any mistakes. In this article I explore this idea in detail.

You may want to think about rewording this, since it is not clear that "reducing the feedback cycle" actually "minimizes the risk of misunderstanding". There is a clear distinction between "reducing the risk of misunderstanding" in communication (especially if we view it as the risk of misunderstanding per cycle) versus "reducing the cost of addressing any mistakes". This is the difference between quality of communication versus timely communication, and I think *quality* communication more so reduces the risk of misunderstanding and *timely* communication reduces the cost of fixing mistakes. It is correct to say having *more* cycles reduces the risk of misunderstanding, simply because if a misunderstanding is not caught on one cycle, it may be caught on a later cycle, so it is reasonable that the more later cycles you have the more likely the misunderstanding, defect or necessary change will be noticed by someone.