I've got a considerable challenge at the moment, and I'd be interested in suggestions and experience from the folks here.
Where I currently work we have a generally agile process. There's a big "pot" of bugs and feature requests, a few folks interact with customers and company strategic decision-makers to prioritise these tasks and assign them to releases. The development team then picks off these tasks roughly in order of release, then priority within the release. For each task we use a TDD approach with occasional pair programming and mandatory pre-checkin reviews. A continuous build system runs each checkin and overnight.
This all works pretty well. We have a theoretically "shippable" product almost every day, and tracking progress is relatively easy. The team has a good sense of collective ownership and people get to cross-train on various areas of the application code and third-party stuff it uses, as well as developing strong skills in specific areas.
Where this process doesn't work so well, though, is in "sharpening the saw". The focus on continually adding customer value seems great, but (to my mind) each task takes much more time and effort than it need to. Even a simple one-character change requires a 20-40 minute rebuild-and-retest cycle, and the Anttest build spews out so many thousand lines of chatter that it's almost impossibe to spot actual errors. the end result is that it feels like TDD with a ball and chain. We are also getting into a tangle with managing client special features and customisations. We have a single "trunk" build, but generating a delivery for any particular customer is a complex, tedious and increasingly fragile manual process.
I reckon we need to include some "sharpening the saw" in our daily and weekly work to improve general productivity. We need to speed up the build process, enable proper testing direct from the IDE, and revisit the architecture of the application to make configuring and managing options for different customers practical.
However much I feel in my gut that such changes will soon pay off in increased productivity and reduced errors, none of them deliver obvious customer value right now. To a client or company strategist they look a lot like classic "spend some time on the infrastructure without delivering value" tasks which Agile approaches are supposed to avoid.
I realize that in an ideal world we would never have got to this point in the first place. The build/test cycle would have been kept fast from day one, and the architecture would have evolved to support client special requests as soon as we got the second one. Given that this has not happened, I'm really interested to hear how anyone else has handled (or might handle) this situation.
We actually negotiated some overhead for this kind of thing with our customer. A Velocity Tax if you will. "We think sustaining x points per iteration is going to get us into trouble in the long run. We'd like to cut our expectations back to x * 0.9 for a while." Since we didn't fire anybody, the dollar cost per iteration stayed the same and the cost per point went up for that time. We made stories internally for the "sharpening" work, but didn't open them up for negotiation and planning with the customer. Sounds like you're ready to tell a compelling story on the ROI for this investment.
A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
How are you slicing such improvement work? Is it done in small chunks, big chunks, or all at once? Is it all the responsibility of one developer who is excused working on other tasks, or is it shared among the team.
How do you co-ordinate changes to architecture and development tools while people are still using the same infrastructure for regular tasks?
In my experience it's best when you incorporate "sharpening the saw" into the work on features. When you work on a feature, make it a habbit to clean up on the way. Everything you touch should be a little bit better afterwards. In my experience this works exceptionally well for design problems. The advantage is that this way you learn to not produce a mess in the first place. Of course the estimates will increase this way, but that is to be expected. For a good motivation from a business perspective, see the "Ken Schwaber Google Talk" video, minutes 40+.
Where this doesn't work very well for us, for example regarding the build system which we don't touch often enough to maintain momentum, we have a "consolidation budget". Every iteration, we have a fixed amount of time to our disposal that we can work on things that are important to us and don't have to directly deliver business value. We use it for refactoring Ant scripts, fixing bugs that aren't officially scheduled etc.
Well, in fact the amount is only theoretically fixed - how much we actually use it depends on both how well the value-delivering work went, and on acute needs. This blends well with the notion of Slack ( http://www.jamesshore.com/Agile-Book/slack.html ). Kent Beck even argues that a team should only commit to an amount of work they are *sure* they can deliver in the iteration. Typically they will be finished with the work before the end of the iteration. Instead of beginning a new feature (which typically won't be finished in the iteration anyway), the remaining time could be invested in the "consolidation" work.
A similar approach seems to be what some teams call the "gold card" approach. I don't know enough about it to tell you more, but a google search might give some good hits.
Hope this helps...
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus