File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Agile and Other Processes and the fly likes Organizational Patterns of Agile Software Development - Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Engineering » Agile and Other Processes
Bookmark "Organizational Patterns of Agile Software Development - " Watch "Organizational Patterns of Agile Software Development - " New topic
Author

Organizational Patterns of Agile Software Development -

Nakata kokuyo
Ranch Hand

Joined: Apr 13, 2005
Posts: 438
hi, mr.James O. Coplien and Neil B. Harrison, do you mind to describe little bit on Organizational Patterns of Agile Software Development book and what the book focus on ? thank you
James Coplien
author
Greenhorn

Joined: Oct 26, 2004
Posts: 22
Kokuyo-san,

Thank-you for starting the conversation!

There are many ways of thinking about the organizational patterns book and there are many approaches I could use to describe it.

If you are a software developer or software development manager, I would tell you that it is a book about how to build products more effectively. If you are interested in the fashionable movements of software I would point out that, historically, these findings and this work are the foundations of Agile software development (see Beedle's review on amazon (he's one of the authors of the very excellent SCRUM book)). You can think of organizational patterns as being the broad foundations on which other techniques build, each optimized (or perverted, depending on your perspective) for its own context: SCRUM, XP, and others.

If you're an academic, I'd say: Here's a good taste of what the real world out there is like.

If you are a businessman, I would tell you that it is a management book that offers insight on many contemporary, practical management practices that we found in software-intensive businesses, but which often apply to other businesses as well. It is not just the dry and strange theories one finds in MBA texts, but living examples drawn from real contemporary projects.

If you are a broad thinker, I would tell you that this is an anthropological study of the culture of software development in the late 20th and early 21st century, a record of the practices, rituals, language, and structures of successful software development teams. The findings are based on a decade of study of dozens and dozens of software development organizations worldwide. The patterns come from ethnographic techniques based on social network theory: a way of studying the (communication) structure of an organization. The analyses are supported by social network formalisms such as sociograms, sociometrics and sociomatrices. Yet the results are communicated in common language that is directly accessible to everyday practitioners.

We communicate the results as pattern languages: a format for conveying the "architecture" of an organization. Patterns appeal to problems at human scale, issues faced by developers and managers every day. Because they are based in real experience, validated over a decade of research (rather than being thrown together into a book in a few months) they form a trustworthy base for organizational improvement. We are proud to build explicitly on, and incorporate some of the work of, other organizational pattern authors such as Ward Cunningham, Alistair Cockburn, Steve Berczuk, and Brian Foote.

Most of all, it's a book about people and groups, not about technology or process.

Well, that's just a few words to get us started. I hope this overview helps set the stage for your consideration.

-- Cope


James O. Coplien<br /> Object Architect, DAFCA, Inc., Framingham, MA<br /> Visiting Professor of School of Informatics, University of Manchester<br /> <a href="http://users.rcn.com/jcoplien/" target="_blank" rel="nofollow">http://users.rcn.com/jcoplien/</a>
Neil B. Harrison
author
Greenhorn

Joined: May 03, 2005
Posts: 8
Let me add a bit. Cope and I have studied the dynamics of software organizations for over ten years, and we have found many fundamental characteristics of highely successful organizations. We have written these characterstics as a pattern language (actually four pattern languages). That is what you find in the book. You can apply these at many levels in an organization, depending on your particular role in the organization, to improve your organization's effectiveness.

Interestingly, these patterns are the foundational material upon which most agile methodologies are built. (Don't take our word for it, ask Kent Beck, Ken Schwaber, Mike Beedle, and Alistair Cockburn.) We found that roles and communication among these roles are key -- they embody, and sometimes shape -- the culture of the organization. And that makes all the difference.
[ May 04, 2005: Message edited by: Neil B. Harrison ]

Author of <a href="http://www.amazon.com/exec/obidos/ASIN/0131467409/ref=jranch-20" target="_blank" rel="nofollow">"Organizational Patterns of Agile Software Development"</a>
Kishore Dandu
Ranch Hand

Joined: Jul 10, 2001
Posts: 1934
Originally posted by Neil B. Harrison:
Let me add a bit. Cope and I have studied the dynamics of software organizations for over ten years, and we have found many fundamental characteristics of highely successful organizations. We have written these characterstics as a pattern language (actually four pattern languages). That is what you find in the book. You can apply these at many levels in an organization, depending on your particular role in the organization, to improve your organization's effectiveness.

[ May 04, 2005: Message edited by: Neil B. Harrison ]


Are u indirectly implying that successful organizations are using agile approaches to create successful projects(unintentionally). But my thinking was, we have agile nominclature that came to prominence only recently. But I did observe many successful projects that employed RUP or waterfall in some cases and not necessarily agile.


Kishore
SCJP, blog
Neil B. Harrison
author
Greenhorn

Joined: May 03, 2005
Posts: 8
Originally posted by Kishore Dandu:


Are u indirectly implying that successful organizations are using agile approaches to create successful projects(unintentionally). But my thinking was, we have agile nominclature that came to prominence only recently. But I did observe many successful projects that employed RUP or waterfall in some cases and not necessarily agile.


No, I'm implying more the opposite; more what you articulated in your second and third sentences. Successful agile projects use many of the patterns that we identified in successful projects of all types. For example, just to pick one or two patterns, Unity Of Purpose is at the base of nearly every successful project. Scrum meetings help reinforce it, if it is already there. Developer Controls Process is an unwritten tenet of many agile organizations, but greases the wheels of any organization. And Pair Programming was actually published in 1994 in an early version of this work (in "Pattern Languages of Program Design" v.1), and its use far predates that.
James Coplien
author
Greenhorn

Joined: Oct 26, 2004
Posts: 22
Originally posted by Kishore Dandu:


Are u indirectly implying that successful organizations are using agile approaches to create successful projects(unintentionally). But my thinking was, we have agile nominclature that came to prominence only recently. But I did observe many successful projects that employed RUP or waterfall in some cases and not necessarily agile.


Kishore,

Of course you are right -- kind of.

The nature of most software projects is that they require an agile approach. Most software development is a human activity full of uncertainty. And there are some hard reasons that waterfall in its purest form must incur high cost. The reason is that, during the process of design, it is fundamentally impossible to know the requirements until you start building something. This is well-established in practice and in the literature (I can send you citations off-line).

In the apparent success of waterfall projects I would guess that, most of the time, you are seeing one of the following:
  • <em>They say they are doing waterfall but if you look under the hood, they're not.</em> My earliest research on ISO 9000-certified waterfall projects bore this out, and it was a finding that we substantiated again and again over the years. People cheat -- and that's a good thing. So Neil and I say: instead of saying we're doing one thing, and then cheat most of the time, why don't we just document what it is we actually do that works? Waterfall is a bad fit for empirical practice most of the time while agile fits. See this ICSP2 paper.
  • <em>Some projects do waterfall as a quality hedge, and they do "work" albeit at tremendous cost.</em> If you look at most aerospace projects with high CMM ratings, they done by a small team under onerous conditions that would tank most commercial software projects. For those teams, which are building life-critical software and done on a contract probably awarded to the lowest bidder, some patterns were probably impossible -- like Domain Expertise in Roles. On the other hand money wasn't an object. It raises the image in my head of unlimited unskilled workers with unlimited money, and it ain't pretty. That can work at very large scales (thousands or tens of thousands of programmers) but not on a project team.
  • <em>"Success" may not be what you think it is</em>. If a project uses waterfall and delivers, I guess it is a success. If it costs ten times as much as an agile project would have cost, I am not interested in that kind of success. In most cases projects have proven that they can develop in an agile mode and deliver more quickly and accurately than analogous projects administered under waterfall. Boehm came to the same conclusion and advocates spiral.
  • <em>Many of these projects have some uncharacteristic advantages over typical projects.</em> Waterfall works particularly well if you're re-doing the same thing that you've already done before. If you already know requirements -- because you've actually built one of the things -- then you know what you need to know to avoid the upstream loops. But if that's the case, waterfall doesn't really help you: it's just that it doesn't hurt you as much as it does in a greenfield design. Pound for pound it's still probably more expensive than the analogous agile development would be.

  • Perhaps I should drive home the first point a little more emphatically. I will say -- and I'm pretty sure that Richard Gabriel has said -- that I have never seen a non-trivial project successfully work forward from requirements to an implementation using waterfall techniques. Many projects say that they do -- but that is usually the manager talking, or the process weenies, and a quick check with the engineers easily invalidates those myths. I have never seen a project without some element of the agile practices in it. Even if you can name the odd exception, I'll bet you can count such exceptions on a very few fingers.

    Since just about everything can be said to be part of RUP, I am not sure what meaning to ascribe to the statements "RUP works" or "RUP doesn't work." RUP seems to be both everything and nothing at the same time. I'll let someone else take up that point.
    [ May 04, 2005: Message edited by: James Coplien ]
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by James Coplien:
Success" may not be what you think it is. If a project uses waterfall and delivers, I guess it is a success.


Often it's even: if it delivers what was called for in the requirements documents, it's called a success. Wether it delivers what the customer actually *needs* at delivery time is secondary... :roll:


Since just about everything can be said to be part of RUP, I am not sure what meaning to ascribe to the statements "RUP works" or "RUP doesn't work." RUP seems to be both everything and nothing at the same time. I'll let someone else take up that point.


Rational even had an Extreme Programming plugin downloadable for its RUP software...


The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
Jim Bracks
Ranch Hand

Joined: May 04, 2004
Posts: 42
To James and Neil

Does the book include any detailed case study of some successful project?

Thanks

Regards
Jim
James Coplien
author
Greenhorn

Joined: Oct 26, 2004
Posts: 22
Originally posted by Jim Bracks:

Does the book include any detailed case study of some successful project?


Sure does, Jim -- two whole chapters. One chapter concerns Quattro Pro for Windows. That project, and the ensuing Dr. Dobb's article I wrote, were a shot heard round the world. People realized they didn't need to cater to onerous management practices any more, and the Agile revolution was born. So as Linda Rising would say, "they were Agile before Agile was cool."

The second case study in the book is of an extremely high-productivity project that was done at AT&T (I don't know if I'm supposed to say that or not, but there you go) which we analyzed for the presence of patterns.

At the beginning of each chapter there is a sequence drawn from a real project. Not all of these sequences have rosy outcomes, but that's the real world. We're not in the business of pulling the wool over peoples' eyes. But we will and do tell you what really can, does, and did happen.

There is another kind of case study you might be looking for, which is: <em>Can I use these patterns to turn an organization around?</em> The answer is: Yes; and we have done it countless times; but not many of these have been written up. One project where the patterns played a crucial role was in the turnaround of ParcPlace Systems (before it merged with Digitalk and tanked). You can find that written up in a JOOP article by Richard Gabriel (who was vice president of engineering there at the time:

<pre>
@article{DBLP:journals/joop/Gabriel94,
author = {Richard P. Gabriel},
title = {Productivity: Is there a Silver Bullet?},
journal = {JOOP},
volume = {7},
number = {1},
year = {1994},
pages = {89-92},
bibsource = {DBLP, http://dblp.uni-trier.de}
}
</pre>

and in his book, <em>Patterns of Software: Tales from the Software Community</em>

Remember, too, that each pattern is supposed to be grounded in at least three examples, and many of our patterns relate anecdotes from our dozens of studies in real companies like Alliance, Navision, AT&T, Schlumberger, and others -- some named, and some not.
Jim Bracks
Ranch Hand

Joined: May 04, 2004
Posts: 42
Thanks James for your detailed reply.
Alvin chew
Ranch Hand

Joined: Jan 08, 2004
Posts: 834
as author view of you , what are the best organization pattern in current day ?
Neil B. Harrison
author
Greenhorn

Joined: May 03, 2005
Posts: 8
Originally posted by Alvin chew:
as author view of you , what are the best organization pattern in current day ?


Well, as you can imagine, it depends on the culture of your own organization right now. Different organizations need different patterns, and perhaps more significantly, are ready for different patterns. The good thing is that most of the patterns can be applied one at a time.

That said, here is a list of ten important patterns:

Architect Controls Product
Domain Expertise In Roles
Function Owner and Component Owner
Engage Customers
Unity of Purpose
Distribute Work Evenly
Firewalls
Mercenary Analyst
Architects Also Implement
Developer Controls Process

In the interest of space, I've just included the names. Holler if you want explanations of any or all of them.
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
I think I can imagine what most of them are about. But I'm not sure about

Domain Expertise In Roles

and

Function Owner and Component Owner

although both sound interesting (and relevant to something my team is currently struggling with). So if you could please elaborate on those. Thanks!
James Coplien
author
Greenhorn

Joined: Oct 26, 2004
Posts: 22
Originally posted by Ilja Preuss:
I think I can imagine what most of them are about. But I'm not sure about

Domain Expertise In Roles

and

Function Owner and Component Owner


Domain Expertise in Roles:
If you need to staff all roles, it's difficult to determine how to match people to roles to optimize communication. Then: match people to roles based on domain expertise, and emphasize that people play those roles in the organization.

Some of the forces (considerations) are:
  • College grades, professional tests and other methodologies tend to work poorly to match people to jobs (http://users.rcn.com/jcoplien/MBTI.pdf)
  • Successful projects tend to be staffed with people who have already worked on successful projects.
  • Spreading expertise across roles complicates communication patterns. It makes it difficult for a developer or other project member to know who to turn to for answers to domain-specific requirements and design questions.


  • Function Owner and Component Owner: If you organize teams by components, functions suffer, and vice versa, Then: Make sure every function has an owner, every component has an owner.

    Some considerations are these:
    • If you organize teams by components, functions suffer, and vice versa.
    • You may be organized by function or use case, with no component ownership.
    • On the other hand, you may be organized by class or component, with no function or use case ownership. In either case you the the same question: "What happens when two people need to program the same function?"
    • You want ownership and consistence in the functions, but also in the components.
    • The components must be shared across the teams.
    • If you just assign function owners, the components become shared, and lose their identity and integrity. But if you have only component owners, the functions become orphans. Who makes sure they get done?


    • [ May 09, 2005: Message edited by: James Coplien ]
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by James Coplien:
Domain Expertise in Roles:
If you need to staff all roles, it's difficult to determine how to match people to roles to optimize communication. Then: match people to roles based on domain expertise, and emphasize that people play those roles in the organization.


I don't think I understand what that means, sorry. Can you perhaps elaborate by explaining a small example? Thanks!

Function Owner and Component Owner: If you organize teams by components, functions suffer, and vice versa, Then: Make sure every function has an owner, every component has an owner.


If I understand that correctly, for example in an XP team we would have owners of user stories, who make sure that a feature is implemented in a coherent way (regarding, for example, usability); and owners of software components (e.g. the persistence layer), who feel responsible for the fact that the code of the components remains coherent?
James Coplien
author
Greenhorn

Joined: Oct 26, 2004
Posts: 22
Domain Expertise in Roles means that for any key competency, there is a person who embodies it. This encourages a culture of ownership, pride, identity and accountability. It also nurtures excellence through specialization rather than trying to make everyone a generalist. Too many methods cater to managers' fears of losing key people, so few methods count on, encourage, or even allow domain excellence. Based on what we've found empirically, they're wrong.

Role normatization is a key component of culture. Culture exists to reduce the energy necessary to get anything done, particularly in high-context cultures. Specialization and well-identified loci of expertise reduce the time it takes to find the oracle to guide you to the end of a bug or completion of a feature. It allows effective pairwise interaction, which is better than ineffective pairwise interaction or frequent n-ary interactions.

Your understanding of Function Owner and Component Owner is correct, but is contrary to what I hear from the Extreme High Priests like Kent. To them there is only corporate ownership; to me, something that is owned by everyone is owned by no one. If someone owns a user story, that's fantastic!
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by James Coplien:
Your understanding of Function Owner and Component Owner is correct, but is contrary to what I hear from the Extreme High Priests like Kent. To them there is only corporate ownership;


In both "Extreme Programming Explained, 1st ed." and "Planning Extreme Programming", individuell developers sign up for and estimate tasks in the iteration planning meeting, and are responsible for the implementation.

Current advice seems to go more along the way of totally dropping tasks from the plan and simply use small enough user stories instead. So we'd actually have individual ownership of stories.

Some teams report that they dropped ownership - a pair out of work just gets the next task/story from the currently scheduled stack. They seem to do well, but it's probably not something for everyone, and certainly not mainstream.

Your impression is more right when it comes to code (component) ownership. It's important to notice, though, that the intent of Collective Code Ownership *is not* to have everyone be equally knowledgable about every part of the code - it is expected of humans to develop interests in specific areas. It's just that it isn't formalized, and noone is hold from gaining knowledge or contributing decisions about some code just because someone else owns it.

to me, something that is owned by everyone is owned by no one.


With all due respect, but perhaps that's only true for you because you think it is true?

Seriously, I think that it's possible to build a culture of quality where everyone feels responsible for (and wants to be pride of) the whole system. In fact I think that there are always some team members who have a finer nose for component quality than their coworkers, and I'd rather have them feel responsible for the whole system - and possibly even produce some peer pressure - than having them only care about their own personal niches.
James Coplien
author
Greenhorn

Joined: Oct 26, 2004
Posts: 22
Originally posted by Ilja Preuss:


With all due respect, but perhaps that's only true for you because you think it is true?

Seriously, I think that it's possible to build a culture of quality where everyone feels responsible for (and wants to be pride of) the whole system. In fact I think that there are always some team members who have a finer nose for component quality than their coworkers, and I'd rather have them feel responsible for the whole system - and possibly even produce some peer pressure - than having them only care about their own personal niches.


That leads to good feelings and good morale, but it's not conducive to action or to business goals--at least, that's what we found over the ten years of studying stuff like this. We found a high correlation (no surprise) between corporate ownership cultures and consensus cultures. We usually found these in two places: 1. Young, idealistic organizations, usually comprising people new to the field, or people who were trying to do something extreme for its own sake, and 2. In the Nordic countries.

The usual failure modes of the first group included analysis paralysis, design paralysis, and most importantly, a lot of "stepping on each others' toes." Tasks became blocked because the synchronization points (the code) weren't owned. Or, worse, they would try to do two things in the same piece of code at the same time, mutually unaware. Pair programming would have been ideal if they had known about each other, but there was no oracle to consult about activity on that piece of code.

So I feel our results are right because we've seen it.

BTW, in the case of Scandanavian projects, there are values in the indigenous culture in which these developments are embedded that bode for their success. These cultures are rooted in a highly egalitarian value system, and that seems to allow ownership to flow. In Norway (where we worked with several companies including Schlumberger), or in Denmark (Navision, Nykredit, and others) it is not a good thing if you stand out or have any personal hallmark of excellence. Reflecting on it, the time cost was high relative to ownership cultures, though; they had more meetings, and more meetings of broader scope, than we tended to find in ownership cultures. (I have a German friend who went from Karlsruhe to Linkļæ½ping and the consensus culture was driving him nuts. Of course, you Germans are another story, too :-) :-) )

Maybe there is something in XP that is reminiscent of the Nordic cultures. I haven't seen anything in XP that screams out for excellence. It's kind of an Animal Farm where all the animals are equal--though the Consultant is a bit more equal than others.

In spite of the fact that the original XP denied the value of either architecture, or ownership, or specialization, or documentation, it is good to know that the "Explained" book took the Code Ownership cue from the pattern language and reversed his earlier position. Even before that book, we, too, had found that most projects that called themselves "XP" had ownership which was, at the time, contrary to The Dogma. I had always found the Extreme Priests to be adamant against it. In fact, it has been interesting to watch XP principles and practices slowly converge on the Organizational Patterns over the years with little or no acknowledgment of the source. (In fact, I know only of one such acknowledgment: http://oopsla.acm.org/oopsla2002/fp/files/spe-metahpor.html, quite a few years into the XP phenomenon.)

Maybe one of the High Priests and Neil and I can write an article together some time about the convergeance. That would be a hoot.
[ May 11, 2005: Message edited by: James Coplien ]
Don Stadler
Ranch Hand

Joined: Feb 10, 2004
Posts: 451
James, I would like you to expand further on the "Function Owner and Component Owner" and the "Mercenary Analyst" patterns.

Also I wonder whether "Firewalls" means well-defined interfaces bewtween subsystems (as I presume) or something else....

Does "Developer Controls Process" mean flexible agile process where ichdevelopers & designers select the artifacts which are produced. Thus getting away from the straitjacket which CMM seems to impose all too often?
James Coplien
author
Greenhorn

Joined: Oct 26, 2004
Posts: 22
Originally posted by Don Stadler:
James, I would like you to expand further on the "Function Owner and Component Owner" and the "Mercenary Analyst" patterns.


I think I covered the former pretty well above: re-read the description carefully and I think you'll understand. The point is that if no one owns a component, there is no oracle to go to who can either tell what is going on with that component at the time or who can coordinate efforts on that component. Someone needs to be around to advocate a component's stake in the architecture, much as the holder of a CRC card does in a CRC card exercise. And if no one owns a feature, the feature may languish. Someone must advocate the feature and champion the customer interest. If you leave this to everyone, you usually have a lot of opinions that are not grounded with customer contact. If you leave all the features to one person, you eschew the domain expertise necessary to represent the interests of the feature to the architecture and the team. So you need both.

The Mercenary Analyst is, as Paul Chisholm relates, the "hired gun" who comes into town, understands your design, writes it up, kisses his horse, saddles up his girl, and rides off into the sunset. You don't want your engineers writing internal documentation. First, every line of documentation they write takes time away from writing code. You sell code, not UML diagrams. (If you sell UML diagrams, you have other problems.) Yet you need some documentation either for the next generation or because the Standards Told You So (FCC, FDA, etc.) Second, engineers are the second worst writers in the world: you don't want them doing your documentation. It's like Vogon poetry. No: worse :-)

Also I wonder whether "Firewalls" means well-defined interfaces bewtween subsystems (as I presume) or something else....


No, it's to keep "people who want to help" off the developers' backs. In ParcPlace Systems we found that the marketing manager and customers would come directly to developers and change their priorities, interrupt the "flow" of their work, and otherwise create havoc. You need a manager in the middle to mediate and to temper the crises, panics, and fads of the moment. So this pattern balances Engage Customer (Developers and Customers Should be Friends) and Developer Controls Process (you need a strategic and global view of feature delivery).

Does "Developer Controls Process" mean flexible agile process where ichdevelopers & designers select the artifacts which are produced. Thus getting away from the straitjacket which CMM seems to impose all too often?


Artifacts are products, and as such fall more under the Architect's influence (Architect Controls Product). The Developer chooses when the artifacts are produced at a granularity of time that is below the project-level (or customer-level) radar screen. Leave the fine scheduling to the Developer: Developers can interleave tasks and fill in short time intervals with work almost optimally. Global planning and scheduling is an NP-hard problem, and inevitably leads to wait states in the methodology.

Of course, this separation isn't hard and fast: it's a matter of focus and emphasis, not of formal delineation of responsibility. But it's good to know where responsibility and control fall by default, and as a matter of good enculturation, it's best to go with the default most of the time.

Neil, do you want to elaborate more?
[ May 12, 2005: Message edited by: James Coplien ]
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by James Coplien:
The point is that if no one owns a component, there is no oracle to go to who can either tell what is going on with that component at the time or who can coordinate efforts on that component.


With six developers working together in a war room, you can simply ask a question into the room. For bigger projects that might probably stop working.

Someone needs to be around to advocate a component's stake in the architecture


There might be a clash of different uses of the term "ownership" here. "Collective Code Ownership" does just mean that every pair has the responsibility to improve every line of code if they find a way to do so. Noone is allowed to say "this needs improvement, but it's not my job". There is nothing in XP prohibiting single developers from being especially competent regarding decisions about specific components.

And if no one owns a feature, the feature may languish. Someone must advocate the feature and champion the customer interest. If you leave this to everyone, you usually have a lot of opinions that are not grounded with customer contact. If you leave all the features to one person, you eschew the domain expertise necessary to represent the interests of the feature to the architecture and the team.


Remember that XP calls for

- the customer describing features at the start of a (one to three week) iteration in the iteration planning meeting,
- the customer specifying the features in automated acceptance tests that must pass the end of an iteration,
- the developers tracking and reporting status at daily stand up meetings, where the customer is present, and
- the customer being available on site for questions during an iteration.

I'm not sure how a feature could languish in such an environment...
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by James Coplien:
That leads to good feelings and good morale, but it's not conducive to action or to business goals--at least, that's what we found over the ten years of studying stuff like this.


Well, that doesn't match my experience at all; but perhaps we are simply speaking of different things here.

We found a high correlation (no surprise) between corporate ownership cultures and consensus cultures.


The book "Facilitator's Guide to Participatory Decision-Making" makes the point that there is a big difference between an inclusive solution and a compromise. I suspect that that difference is very similar to "collective ownership" versus "no ownership".

The usual failure modes of the first group included analysis paralysis, design paralysis


Was that because they couldn't find consensus on what the result of the analysis/design should look like?

and most importantly, a lot of "stepping on each others' toes." Tasks became blocked because the synchronization points (the code) weren't owned. Or, worse, they would try to do two things in the same piece of code at the same time, mutually unaware. Pair programming would have been ideal if they had known about each other, but there was no oracle to consult about activity on that piece of code.


Sounds like they didn't practice Continuous Integration. I'd also wonder why they'd need an oracle to know who was currently working on what. In the team I'm working on I typically happen to know, and if not, I ask. Perhaps a matter of team size again?

So I feel our results are right because we've seen it.


I don't dispute that you are likely to have seen that a lot.

But I think that XP has different solutions to offer to the same problems, solutions that might be worth a look. And I know that it can work, because we are very close to doing it, and in my opinion quite likely would do better if we were even more close (to doing XP).

These cultures are rooted in a highly egalitarian value system, and that seems to allow ownership to flow. In Norway (where we worked with several companies including Schlumberger), or in Denmark (Navision, Nykredit, and others) it is not a good thing if you stand out or have any personal hallmark of excellence.


That is nothing that XP enforces, though. A coworker of mine is known for excelling at GUI design. When someone has finished a new non-trivial dialog, he gets consulted to suggest improvements. Another coworker is our database specialist and he too gets the praise he deserves. But that doesn't mean that he isn't responsible for improving GUI code when he sees potential for improvement.

Reflecting on it, the time cost was high relative to ownership cultures, though; they had more meetings, and more meetings of broader scope, than we tended to find in ownership cultures.


It's not obvious to me that more meetings automatically mean less effectiveness.

Maybe there is something in XP that is reminiscent of the Nordic cultures. I haven't seen anything in XP that screams out for excellence. It's kind of an Animal Farm where all the animals are equal--though the Consultant is a bit more equal than others.


The XP you know must be significantly different from mine. I wonder where that is coming from...

In spite of the fact that the original XP denied the value of either architecture, or ownership, or specialization, or documentation


Did it? For achitecture? Documentation?

it is good to know that the "Explained" book took the Code Ownership cue from the pattern language and reversed his earlier position.


As far as I can tell, it's now talking about "Shared Code" as a corollary practice ( http://xp123.com/xplor/xp0502/index.shtml ). I haven't read the book yet, but I doubt that it qualifies as reversing the position.

Even before that book, we, too, had found that most projects that called themselves "XP" had ownership which was, at the time, contrary to The Dogma.


As far as I know, there is no dogma in XP, just practices. There was some time when Kent Beck called them "etudes" (don't know wether he does that any longer).

But I still wonder what exactly you mean by "ownership".


I had always found the Extreme Priests to be adamant against it.


Probably driven by experience, too...

In fact, it has been interesting to watch XP principles and practices slowly converge on the Organizational Patterns over the years with little or no acknowledgment of the source.


I always thought that patterns didn't have a single source, that they were observed in the wild and then catalogued when we found repeated instances of them. I'm quite sure that I used the Memento pattern before I knew that it was called that way in the GoF book...
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Kent Beck on Shared Code: http://groups.yahoo.com/group/xpbookdiscussiongroup/message/662
James Coplien
author
Greenhorn

Joined: Oct 26, 2004
Posts: 22
Ilija -- your long response above leaves me, well, speechless :-)

In the interest of Saying the Simplest Thing that could Possibly Work, let me summarize by saying that our broad research over a decade points to the value of ownership. It doesn't matter as much in simple or general-purpose domains, but it matters in those cases where it matters to have a business edge.

I've come to the conclusion over the past two weeks that XP has finally reached the stage of RUP: it can be anything you want it to be. In fact the RUP people not only have an XP plug-in but claim that RUP is a form of XP. We have studied dozens of so-called XP projects .The only two real XP projects I know of (that follow all the practices (the old version)) -- C3 and Ken Auer's company -- didn't following a thriving trajectory. Pick any practice, or any combination of practices, and you can call it XP. (I don't know if you legally can do so, and whose it is to say whether one can or can't, but the truth is that people do call anything and everything XP. And I haven't seen the consistent set of High Priests who can stick to one story for more than a couple of years or even at the same time. The same is true for just about any high-context method, by the way -- it's not all XP's fault.)

What's interesting is that other high-context cultures we've studied, such as the Borland QPW project that, historically, kicked off the entire Agile movement (including XP) also has an obscure trajectory. I worry a lot about burnout; we have several sections about this in the Organiztional Patterns book. I know XP has an 8-hour day, but the pressure of appearing accountable before a group in a morning stand-up meeting puts developers in a posture where they carry the work home in their head. Pair programming means that there's always a (implicitly, "big") brother looking over their shoulder. <em>Ce grand malheur, de ne pouvoir etre seul -- La Bruyere</em>.

We need more empirical work in this area, more substantiation of what does and does not work. The rest is just flapping our gums. Neil and I were careful to spend ten years validating our claims. The early claims of XP were contrary to some of those claims (and I need to go back to the first book rather than the modern evolutions you cited earlier to substantiate these -- maybe in my 'blog some day), and XP has indeed slowly been migrating toward the organizational patterns. The latest crisis for XP has been adaptability -- something that was in the pattern vision way back when the High Priests were still in pattern land and before XP had been created as a marketing buzzword.

Long ago a poor frustrated grad student approached me, a lone software engineer in the middle of a onerously formal CS department, looking for a Ph.D. thesis topic. I suggested doing some empirical work on this newfangled thing called XP and, in particular, on the intriguing notion of pair programming that it had adopted from long past history. Laurie Williams took up the task eagerly and became the darling of the XP community. In her classroom experiments it looked like a big win. However, I am always skeptical of classroom experiments and suggested she do a study with one of my clients in Europe. The clients later let me know that the results were counterindicative to the value pair programming. Yet an article recently appeared here about the value of pair programming in a military project. I've worked with military projects on three continents and can believe it works there, in a high-context culture where discipline is handed out in bottles the first day you arrive on the job.

Software development is a collection of largely undisciplined cultures -- yet the words "discipline" and "culture" are almost oxymoronic. Cultures are what they are. Studying them even through qualitative methods is fiendishly difficult. I think we will see the patterns only through longitudinal studies. (The organizational pattern work is the only study I know of that looked at software process over a period of longer than a year, and we took ten years.) The results of these studies will vary radically from culture to culture, organiztaion to organization, and product to product. Don't believe what's in the books. Unlike books written by most consultants, the bottom line of patterns isn't to give you a method to follow, but to open you up to the opportunities provided by your own instinct. That was Alexander's whole purpose -- and when I invited him to speak to us at OOPSLA in 1996, he sadly noted that we had missed the boat. He noted the organizational patterns as an exception. There is something worth pondering there.

Of course the market is the ultimate vote. But the market is wrapped up in our identity. Do we want to cater to market forces and do the simplest thing that people would put down money for so we could get rich, or do we take up the moral imperative to make the world a better place? In the long, long term, the market won't have the final vote: the Tao will, as will our consciences. That means working together as a team, working in such dialectic as we find here -- I rejoice at the opportunity to bump ideas up against Ilija -- rather than fragmenting our efforts in industrial competition. It means standing on each others' shoulders, understanding each others' work, instead of standing on our tiptoes and re-inventing the wheel. I know there is still much I can learn about the "theory" of XP. But I have invested in understanding its practice, and I stand on that experience. It is not an absolutist stand, because different things are, well, different.

By the way, after today I'll be leaving the ranch and headed back into the rural Danish countryside where the Hildeblomster will soon be in bloom, anticipating the warm winds of summer that their nectar will cool. If any one wants to carry on the fireside chat outside the saloon and beyond the ranch, you can reach me at J O C o p l i e n - a t - c s . c o m. I'd love to meet up with you there.
[ May 20, 2005: Message edited by: James Coplien ]
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by James Coplien:
Ilija -- your long response above leaves me, well, speechless :-)


Fortunately you were still able to write...

In the interest of Saying the Simplest Thing that could Possibly Work, let me summarize by saying that our broad research over a decade points to the value of ownership.


I would be highly interest in details - especially in the definition of "ownership" you use(d).


It doesn't matter as much in simple or general-purpose domains, but it matters in those cases where it matters to have a business edge.


Just for the record, I'm working on an application that I would call far from being simple, and which depends heavily on innovation to be successful.

I've come to the conclusion over the past two weeks that XP has finally reached the stage of RUP: it can be anything you want it to be.


What exactly is bothering you about this? (Not that I fully agree, but let's put that aside for now.)


What's interesting is that other high-context cultures we've studied, such as the Borland QPW project that, historically, kicked off the entire Agile movement (including XP) also has an obscure trajectory.


I guess you are referring to http://www1.bell-labs.com/user/cope/Patterns/Process/QPW/borland.html ? I wasn't aware of this project (at least I don't remember it), will take a deeper look tomorrow...


I worry a lot about burnout; we have several sections about this in the Organiztional Patterns book. I know XP has an 8-hour day


Ron Jeffries used to call it "Sustainable Pace", Kent Beck now renamed it to "Energized Work". He also newly included Slack as a primary practice.

the pressure of appearing accountable before a group in a morning stand-up meeting puts developers in a posture where they carry the work home in their head.


Not in my experience. The purpose of the stand up is to identify and resolve issues early; if at all, it should *reduce* pressure on the developers, because it provides them a platform to get help.

I also don't see anything wrong with being accountable for my work - at least now that I followed the recent, heated thread about it on the XP mailing list and finally think to understand how Kent Beck uses the term...

Pair programming means that there's always a (implicitly, "big") brother looking over their shoulder.


If your programming partner feels like a "big brother" to you, you are doing something wrong. In PP, the partners are supposed to be, well, partners.

We need more empirical work in this area, more substantiation of what does and does not work. The rest is just flapping our gums.


I'm not sure what you are needing this for, but while you try I will happily follow anecdotal reports in the various forums and try wether things work *for me*.

Doesn't mean that I won't be interested in the "substantiation" once it becomes available. But I will certainly trust my own experience more than any empirical work, as you will hopefully understand...

However, I am always skeptical of classroom experiments


Actually I'm always skeptical of experiments. I just recently participated on a non-classroom experiment on test driven development, and that didn't really improve my confidence in any academic studies.

The clients later let me know that the results were counterindicative to the value pair programming.


Interesting. Is there a way to learn more about this? How much PP coaching did they receive, for example? How did PP interact with the other practices of the team? What values did the team have? Etc. pp.

I've worked with military projects on three continents and can believe it works there, in a high-context culture where discipline is handed out in bottles the first day you arrive on the job.


Mmh, all reports I've read actually seem to indicate that about 90% of all programmers *enjoy* PPing, after some time getting used to it.

Software development is a collection of largely undisciplined cultures


I'm not sure I understand what you are trying to say - do you care to elaborate?

(The organizational pattern work is the only study I know of that looked at software process over a period of longer than a year, and we took ten years.)


How does XP (just as an example) compare? Wasn't it also first discussed at the wiki? Wasn't it continuously discussed, at different places, since then? Couldn't perhaps the second also be seen as a result of observing XP being used and discussed for a decade?

The results of these studies will vary radically from culture to culture, organiztaion to organization, and product to product. Don't believe what's in the books. Unlike books written by most consultants, the bottom line of patterns isn't to give you a method to follow, but to open you up to the opportunities provided by your own instinct.


That's how experienced developers are using XP, too, in my experience. As discussed above, it won't work well for people new to XP or Agile development.

Do we want to cater to market forces and do the simplest thing that people would put down money for so we could get rich, or do we take up the moral imperative to make the world a better place?


Well, I'm certainly more driven by the latter. Interestingly, many people report that XP brought back the joy to software development for them.

Of course I wouldn't object to getting some money, either. I'm still waiting to get rich, though.

I rejoice at the opportunity to bump ideas up against Ilija


It's certainly a thought provoking discussion!
Don Stadler
Ranch Hand

Joined: Feb 10, 2004
Posts: 451
Originally posted by Ilja Preuss:

James Coplein wrote: "I've come to the conclusion over the past two weeks that XP has finally reached the stage of RUP: it can be anything you want it to be."

Ilja: What exactly is bothering you about this? (Not that I fully agree, but let's put that aside for now.)


I cannot speak for James, Ilja, but I've worked at several projects which purported to follow RUP. Most of them were thin wrappers on some other process - often waterfall or at best COCOMO (waterfall with iterative wrapper). Some time in bars with some Rational people convinced me that the intent of RUP is rather more agile than that. What I saw in those projects was a degenerative case. But the degenerative case seems to be the norm with RUP.

I suspect James was making the same point about xP, that he suspects something like that is happening with projects purporting to be xP. My personal experience with a single psuedo-xP project was an unhappy one, with a totally mishandled SCRUM being the major cause of trouble. You would possibly retort 'that wasn't xP'! And you would be correct. But if 9 out of 10 xP projects are actually a degenerative form of xP, that is what people will identify as xP.

I think real xP can be very good - but a degenerate form of xP can easily turn into a tyranny when run by a control freak. SCRUM can turn into a regular morning inquisition rather than the brief informational meeting it is intended for.

I can't guarantee that my PM won't be a power-drunk sadist, so unless I knew them very well beforehand I would tend to be suspicious of doing xP/SCRUM with a manager I didn't know. Because SCRUM strips away all you're defenses - and that can be taken advantage of.

Another thing to consider. I suspect that you're primo inter pares (first among equals) in your team. The viewpoint of how well a methodology is working can vary a great deal between team leaders and the foot soldiers. If you have true workplace freedom or democracy they are probably Ok with it. But if you don't they may not tell you about it. How is your turnover rate - that can be a strong indicator of discontent?
[ May 23, 2005: Message edited by: Don Stadler ]
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by Don Stadler:
I cannot speak for James, Ilja, but I've worked at several projects which purported to follow RUP. Most of them were thin wrappers on some other process - often waterfall or at best COCOMO (waterfall with iterative wrapper). Some time in bars with some Rational people convinced me that the intent of RUP is rather more agile than that. What I saw in those projects was a degenerative case. But the degenerative case seems to be the norm with RUP.


I guess the lesson to learn from this is that people will call their process whatever they want, wether it is "officially allowed" or not. Isn't it?


I suspect James was making the same point about xP, that he suspects something like that is happening with projects purporting to be xP.


It was happening at the very beginning of XP (which actually drove the community to become more "dogmatic", as far as I can tell). I don't think that in this respect, it really matters much what Kent Beck "allows" or doesn't "allow".

But if 9 out of 10 xP projects are actually a degenerative form of xP, that is what people will identify as xP.


Perhaps people should care less about wether they are "doing XP", and more about wether what they are doing is appropriate to their situation? On the other hand, quite possibly the person in charge thought that it was appropriate. He probably didn't understand the practices, because he didn't share the *values* of XP. Do you think that "doing full XP" would really have made a difference?

I think real xP can be very good - but a degenerate form of xP can easily turn into a tyranny when run by a control freak. SCRUM can turn into a regular morning inquisition rather than the brief informational meeting it is intended for.


Agreed. Again, I think this is more a matter of a different value system than a matter of doing all the practices.

I can't guarantee that my PM won't be a power-drunk sadist, so unless I knew them very well beforehand I would tend to be suspicious of doing xP/SCRUM with a manager I didn't know. Because SCRUM strips away all you're defenses - and that can be taken advantage of.


Frankly, I wouldn't want to work anywhere I felt that the process had to protect me from the people whichs basic goal actually should be to support me. When in such a situation, I would work on "changing the organization, or changing the organization", regardless of the process.

Another thing to consider. I suspect that you're primo inter pares (first among equals) in your team.


No, I think I can say with confidence that that's quite certainly not the case.

We regularly discuss process issues in the team, most often informally. And though I'm often the one who suggests specific practices, I typically don't have to lead the effort - actually I think that two of my coworkers are much better than me in actually getting the ball rolling. I'm learning from them, though...


The viewpoint of how well a methodology is working can vary a great deal between team leaders and the foot soldiers. If you have true workplace freedom or democracy they are probably Ok with it. But if you don't they may not tell you about it. How is your turnover rate - that can be a strong indicator of discontent?


Zero, for the last 3,5 years (when I joined the team).

There currently *is* some discontent, though, which has to do with the changes in the organization I discussed in another recent thread.

One of the complaints of the most discontent team member, a good friend of mine, is that she couldn't pair program for the last weeks.

Another coworker, who was known as the "lone wolf" three years ago, is pushing hard for pair programming - he is even getting on my nerves from time to time... Just yesterday he asked me wether he could do anything to improve the experience for his partner.

And the fourth and last core developer also enjoys pair programming, although he also likes to "single program" from time to time. We even did a "pair presentation" on an XP conference last year, and plan on repeating it.
Stan James
(instanceof Sidekick)
Ranch Hand

Joined: Jan 29, 2003
Posts: 8791
I think about the only value of knowing "what XP is" would be when meeting strangers: "We do XP", "Ok, I know 12 things you do exactly like Kent". And since nobody does 12 things exactly like Kent, that value is pretty well wiped out. A lot more people can say "We do some XP practices within the code production part of something like SCRUM, except our planning game is different" which opens a conversation instead of ending one.


A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by Stan James:
which opens a conversation instead of ending one.


Don Stadler
Ranch Hand

Joined: Feb 10, 2004
Posts: 451
Agreed. Again, I think this is more a matter of a different value system than a matter of doing all the practices.


Excellent. This is the way my thoughts have been going. On my nightmare psuedo-xP project we did most of the xP practices - but the values were way off.

Perhaps one lesson to take from this is that you have to keep a sharp eye on the values - particularly but noy limited to SCRUM?
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by Don Stadler:

Perhaps one lesson to take from this is that you have to keep a sharp eye on the values - particularly but noy limited to SCRUM?


Definitely!

On the other hand, values are rather hard to communicate; and it's quite easy to pay lip service to them. That's one of the things the practices of XP are good for: teaching the values of XP and how they actually *can* be put into action.

I guess it doesn't work for everyone... <sigh>
Don Stadler
Ranch Hand

Joined: Feb 10, 2004
Posts: 451
Originally posted by Ilja Preuss:


Definitely!

On the other hand, values are rather hard to communicate; and it's quite easy to pay lip service to them. That's one of the things the practices of XP are good for: teaching the values of XP and how they actually *can* be put into action.

I guess it doesn't work for everyone... <sigh>


No. Thats the thing. You need to drive out fear, and that is where we blew it. The manager was out to show that xP was a great thing as quickly as was impossible (though he didn't admit it) so he set the goals impossibly high. That is we had the planning meeting - and he stacked the deck. That led to fear, and the morning 2-hour SCUM (sic) grillings engendered more fear. I was the first to speak out and the first to be fired when upper management went headhunting.

The values we're discussing are important to any project - not only xP. I suspect xP may have some of the best outcomes - if you get the values right. Conversely I think xP can be more of a disaster if you get the values wrong. I have been able to get good results even in some very disfunctional situations - because I find cracks in the wall where I can do good work. Authoritarian xP (psuedo-SCRUM) plastered over those cracks and I couldn't work around the lack of trust.

That is my point, I guess. It's a big win but also a big risk.
Edwin Keeton
Ranch Hand

Joined: Jul 10, 2002
Posts: 214

Ilja, you obviously are a member of a functional, cohesive team. I'm heartened to hear that such a thing really can happen, at least in the right environment.

I'm curious about the role local business practices and customs play in creating (allowing) these conditions where the values of XP can be put into action. Here in the U.S.A., I don't think I've even heard about teams staying together (sans major turnover) for much longer than the lesser period of either a single project, or several months, maybe a year at most. (Which usually are approximately equal anyway.) Might I ask where you work and live?

In my experience, most business organizations have a dominant authoritarian "culture of fear" no matter how hard they might protest, or how hard they might actually be trying not to. It seems to be difficult to "manage" creative and intellectual work (how can I tell whether you're thinking or not) which seems to ultimately produce fear in the manager. This is especially the case in organizations whose primary products or services are not software.

I think this fear factor is another important force, maybe almost as powerful as the apparently obvious "bottom line" force, behind the outsourcing phenomenon. Put the thing we fear or dislike out of sight.

The agile and adaptive development methods are intended to eliminate, or at least greatly mitigate, management fear by producing frequent, tangible, working results. Sadly, way too often one agile method or another is adopted in a degenerative form. (Degenerative cases do seem in practice to be the norm.) When it necessarily fails, "we tried that and it just doesn't work" becomes a firmly entrenched idea.

Change is hard to institute anywhere, anytime, regardless. "How do I measure the progress of this XP project compared to the one last year, the way we always did it before." "How do I put this XP stuff into MS Project." "How will I know how to allocate 'resources'"? (the favorite term for people) We have our work cut out for us, as Don put it, finding the cracks in the wall where we can actually accomplish good work.


SCJP, SCWCD
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by Don Stadler:
You need to drive out fear


Yes. It took me some while to understand why Courage is one of the (now) 5 values of XP, but the longer I'm in that process thing, the more I think it's really vital.

The manager was out to show that xP was a great thing as quickly as was impossible (though he didn't admit it) so he set the goals impossibly high. That is we had the planning meeting - and he stacked the deck. That led to fear, and the morning 2-hour SCUM (sic) grillings engendered more fear. I was the first to speak out and the first to be fired when upper management went headhunting.


Uh, that sounds truly disfunctional...

The values we're discussing are important to any project - not only xP. I suspect xP may have some of the best outcomes - if you get the values right. Conversely I think xP can be more of a disaster if you get the values wrong.


If I were into the is/is not XP thing, I'd probably say that if you don't get the values right, it certainly isn't XP, no matter what else you practice...
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Organizational Patterns of Agile Software Development -
 
Similar Threads
How do you write an application?
Best Java book you have read this year?
UML for Java Programmers
Agile Adoption Patterns: competing books
* Welcome Amr Elssamadisy