How to prepare for production?

[Mike Farnham]

Welcome Michael!

Based on the brief description, on Amazon, the book appears to focus, on what else needs to be done to the Release 1.0 version before it can be deemed production ready. Where "production ready" means whatever happens after it is rolled out into the Production Environment.

What type of techniques allow for preparing for the Production Environment even though you really can't "Test in Production"?

Cheers,
Mike

[Edit by Dave to provide meaningful topic]
[ February 13, 2007: Message edited by: David O'Meara ]

[Perry McKenzie]

Hi Mike and Michael,

Maybe I can throw in my two cents here...

One technique (if you can call it that) we've found to be particularly helpful to prepare for a production release when you can't really 'test in production' is to have one or more staging environments that mirror the production hardware and network configurations. This enables you to test without worrying about those variables once you've pushed your code to prod. Staging environments set up like this also allow you to load/performance test so you can 'pre-tune' the memory allocation, file handle size, etc. for your servers.

Cheers,
Perry

[Michael Nygard]

Mike and Perry,

Perry, you're definitely on the right track. There are a whole class of release problems caused by the sort of "Does QA match Production?" question. I wish I had a nickel...

The fact that QA environments tend to be tiny, flat environments with no firewalls creates endless hassles.

Yes, you can audit QA against production, but there will always be some number of "known differences"... There have to be: hostnames, cluster configurations, database connection parameters, external integration points, etc. If there were no difference, QA wouldn't be QA. It would be production!

So, there will always be some tolerable deltas. I try to identify which kinds of deltas are likely to cause problems.

For instance, saving a few thousand dollars by eliminating firewalls in QA costs many thousands of dollars in downtime when some firewall rule change gets missed, or a stability problem (like "The 5AM Problem" story I tell) sets it. It's a really foolish way to save a few grand on network gear.

Another kind of discrepancy that can cause rude surprises is in multiplicity. In dev and QA, pretty much every system looks like one or two servers. So system A and B both have roughly balanced capacity. Now what happens when A is deployed across 20 app servers and it makes an XML-over-HTTP call to system B, which is on 2 app servers in production?

Cheers,
-Mike

[Arun Batchu]

Mike,
Usually it is such a trouble to get *just* the functional requirements taken care of, that the very mention of worrying about the "-ilities" that a good Architect would bring up and which are absolutely necessary and become important after the birth of a baby (what do you mean the button is on the next page?), are either deliberately ignored or pooh-poohed. People run out of steam, energy and most importantly budget and/or time just getting through the functional requirement gauntlet that they are made to run through.
Unlike a baby, whom we nurture for years before pushing her out into the wide, wild world, and that too under a watchful eye, we tend to demand our fledgeling systems to be functional, resilient, performant, scalable, secure, ACID, alkali... all that ... with virtually NO nurturing time, money or even thought.

The only way I see a way out is to "figure that quality factors in" upfront into project scope, budget etc. rather than demanding a separate "QA" and "performance testing" budget. I would really love to get some heuristics about how to get those taken care of by matter-of-factness, like scheduling a performance/scalability test or usability tests at every milestone rather than at the final release or "staging" area.

I am tired of being a hero. I need to get a life - the remainder anyway.

Your thoughts?
-Arun.

[Michael Nygard]

Arun,

You make a good point. We don't seem to have enough time for just the functional behavior.

The good news is that most of the recommendations in my book (expressed in the form of patterns and anti-patterns) don't actually take longer to implement. It's sort of like sitting at a fork in the road, faced with two choices that appear equivalent. Take the left fork, and you'll be saddled with the 88% availability that is typical of most J2EE applications. Furthermore, it will be so difficult to change that you want to defer releases until there's "enough" new functionality to make the pain of a release worth it. Of course, there's a vicious cycle there, because the more that goes into the release, the more likely it is to fail... thus making you even more reluctant to release new versions.

Take the right fork, and you'll achieve 99.9% availability and be able to roll out new features every couple of weeks. With your flexibility, speed, and high reliability, your users will become loyal and you will bury your competitors under an avalanche of features.

Building software that survives production makes it easier and cheaper to implement features.

Furthermore, suppose you do make it through to Release 1.0 with software that crashes all the time, can't scale, can't handle the demand, and can't be changed. I guarantee that you and the other developers will spend a large percentage of your time putting out fires in production rather than building new features. That won't be limited to work hours, either. I've lived that scenario many times, and I'd rather not wear a pager for the rest of my life.

All of which is my way of saying, you're more likely to recover your life outside work by delivering production-ready software.

Cheers,
-Mike

[Ilja Preuss]

Originally posted by Arun Batchu:
I am tired of being a hero. I need to get a life - the remainder anyway.

Kent Beck's "Ease at work" talk ( http://video.google.de/videoplay?docid=7830246530742207581 ) might be interesting to you.