Here's an interesting snippet reporting a study by the UK Warwick Business School which claims "unnecessary complexity is costing business over 10% of their profits, equivalent to $237 billion for the Forbes top 200 firms".
I've worked on several seriously over-engineered J2EE applications in recent years, which have cost the end client a bucketload of money in unnecessary bloatware and slow development, and I've seen some fairly robust criticism of old-style J2EE "complexity merchants" from Rod "Mr Spring" Johnson.
So what about the rest of you? Do you see a lot of unnecessary complexity being imposed on business systems? Or do you think the complexity is necessary? Or do you simply ask "what complexity"?
As Einstein is supposed to have said: "Everything should be made as simple as possible, but not one bit simpler."
Its it really J2EE? or did you mean perhaps JEE? In my brief exposure to J2EE a few years back, I ran away screaming and hollering at the baroque complexity.
On the basic topic, I believe that yes, many companies build or have built systems that are much too complex, sometimes orders of magnitude too complex. This is more critically a problem with governments hire consulting firms, as the consulting firms have serious incentive to add complexity and thus add costs. The problem is not just seen in government contracting, its common in other big industries, such as banking and home mortgage, at least before the big credit meltdown.
I am developing a new thesis on this topic. For 25+ years, I worked in large consulting companies building big systems. Things that cost at least a million dollars and often cost many tens of millions of dollars, took several years, and had 100 or more engineers and analysts working on them. I think that such projects may be obsolete and impossible to deliver.
The problem is not that these systems are now too expensive, or that big systems are hard to do -- they have always been expensive and hard. What has changed is expectations. You can buy fairly complex applications for your smartphone for $5. Its hard to see why anything is a million times harder than that. In the olden days, the consultant worked with the customer's IT staff and made decisions. These days, everyone in the whole corporation has opinions on how the new system should be built. While opinions are like bellybuttons, everyone has one, in the olden days, systems were built without listening to everyone, and without trying to keep everyone happy. With a less than infinite budget, you *can not* make everyone happy.
Looks like it's just you and me on this issue. What are we doing wrong?
Yes, I should have said "JEE" rather than J2EE, although my own most painful direct experience of this problem was with J2EE (EJB 1.1). However, I've worked on several enterprise Java projects in the public sector in recent years, and every one of them has suffered from their own variety of unnecessary "baroque complexity" which has cost the client a lot of time and money that could have been better spent delivering greater functionality and benefits to the taxpayer.
Of course, this is not a Java-specific issue, although I do think the shift to Java has tended to reinforce the experimental geek tendency in commercial IT at the expense of using reliable robust tools to just get the job done. All those open source tools and libraries are great, but many places I've worked seem to spend a lot of time trying to bolt them all together from scratch, instead of finding out if somebody has already invented that particular wheel for them, so they end up piling their own homespun complexity on top of the complexity already inherent in the choice of platform.
There also seems to be a lot more of the "Not Invented Here" syndrome around these days, perhaps because the availability of these "free" components encourages people to re-invent their own solutions to common problems. Geeks love a challenge and it can obviously seem like more fun at a personal intellectual level to build your own super-complex solution, but it's not necessarily cheaper for your client. Also, too many organisations and individuals forget that this stuff is hard, and we're not all the genius coders we might like to think we are. Well, I know I'm not.
You're right about the pernicious influence of the big consultancies on government IT especially, and I've seen plenty of gold-plating on these IT projects, and an often complete failure by the government organisation to act as an intelligent customer in these consultancy deals. These guys often display less sense when spending millions on an IT project than they would apply when buying a used car (because they're not paying for the IT project - we are). These projects are invariably plagued by complexity and the resulting escalation in costs for what is often minimal functionality.
Your point about expectations and increased user involvement is interesting - I hadn't thought of that. I wonder if the rise of Agile also plays a role? Agile approaches preach far more user involvement and responsibility, and (at least in my experience) can sometimes tend to focus too much on what the user sees rather than the underlying implementation. I seem to spend a lot of time these days endlessly re-working minor pieces of "visible" functionality to meet user requirements, while simultaneously having to chase them for months to get answers to questions on how they want the underlying system to actually work.
Returning to the car analogy, it's true that as a user, I don't necessarily need to know the details of how my VW works under the hood. But as a customer I certainly want to know that it is reliable, fuel-efficient and will cost less than 3 times my original budget. In other words, a customer has a right to expect software that will do the job at a reasonable cost, but they also have a responsibility to their shareholders or the taxpayer to ask the right questions and make the necessary compromises to achieve this.
I dunno. It looks like there are a lot of parallel trends pushing IT projects down this complexity path: inherent complexity in the technology, greater and perhaps unrealistic customer expectations, too many computer scientists pursuing their own interests at the customer's expense, too many government customers abdicating responsibility for their own part in the process, and too many outsourcers and consultancies who make lots of money from increased complexity and the corresponding development/maintenance costs.
I think I need to brush up on Grails, just to remind myself it doesn't have to be this way!
While parts of the Agile method are good engineering, the whole embrace of the "spiral" development model is a disaster. With spiral/agile, first you write a tiny bit of code and get feedback. If its wrong, you change it, and make it better. Repeat until done.
Raises a lot of questions:
1) when is it done?
2) software engineers love developing new stuff, and they hate maintenance of old crock code. With Spiral, they get to do new development for a week, and then switch instantly to maintenance mode, forever.
3) with spiral, there is no accountability for the code and design. It may have started well, but over time, things change, code changes, and it gets patched.
4) with spiral, there is no reason for the customer to reach an agreement on what they want, what is acceptable. They don't know until they see one round of the spiral, and only then can they throw rocks at it.
I worked for 15 years mainly on RAD projects using proprietary (mostly Oracle) tools. Of course, there were all the usual issues over proprietary licencing costs etc, but in the end we usually got the job done pretty much on time/budget. At the same time, the tools allowed us the flexibility to change things quickly and present prototypes to the user where appropriate. Some of these projects were quite small, others were pretty big systems by the standards of the day, and taking a structured but flexible approach based on relatively frequent small deliveries, and regular user involvement where possible, seemed to work pretty well.
Although we didn't know it then, I guess we were doing a form of spiral development. But we also had certain functional requirements we had to meet, which defined when we were "done", although by the time we got to that point there were more requirements, so we'd start on the next version of the product.
So on the one hand, a lot of the Agile ideology seems fairly obvious to me - we were doing a lot of this stuff years ago, and I've never worked on a classic "waterfall" project in 23 years anyway.
But on the other hand, every self-declared "Agile" project I've worked on in recent years has suffered from exactly the problems you describe. And it's a particular problem with the big serious stuff deep in the guts of the system that the user doesn't see but which you have to get right early on because it will cost a fortune to change it later. I haven't yet figured out how any Agile approach can deal with those issues adequately.
But I think I'll leave it to somebody else to kick off the "Is Agile a failure?" debate - there's way too much ideological heat around that particular issue!
Ulf Dittmer wrote: Since this is a serious discussion, I'll move it to a more appropriate one.
Yeah, its a bit too meaningful for MD. But Ulf, you need to chime in. Please
Joined: Mar 22, 2005
It seems to me that the article Chris mentioned first talks about something other than what is discussed here - the complexity of having to support or integrate several distinct systems, whether they are acquired through M&A or developed to serve different business areas. In my view that's mostly an integration issue, not so much a development issue. There generally is not enough time and manpower to create a coherent single software systems from such diverse apps, nor does that really need to be the case IMO (assuming that issues of data overlap and consistency can be handled).
As to agile/spiral, I'm more with Chris than Pat - such approaches do not mean the end of requirements, or clearly defined works assignments, or customer sign-off, so I don't see the problems Pat mentions in #1 and #4. #2 and #3 are matters of internal organization, but I don't see the issues that arise with agile/spiral approaches as much different than they would otherwise be. Code does not become legacy code or unmaintained between iterations - it is being worked on (and refactored, if necessary) as long as the project goes on. Then it might be said its legacy status begins, and that's the same as with other approaches.
As to the more nitty-gritty stuff - yes, EJB 1 and 2 needed to be corrected by EJB 3. That may pull back some of the folks who wandered off to the Spring camp, now that Spring is quite heavy-weight itself. A similar thing could be observed with remote services - RMI wasn't the answer people were looking for, SOAP was better but became way over-engineered (mostly driven by technology vendors IMO), so now we have REST which works well for many applications.
The underlying trend I see is away from all-encompassing solutions towards a pick-and-choose attitude. JEE's role would be more of a yardstick - knowing that a JEE-compliant server can run all the JEE APIs, I would still just use those that make sense to a given project. More often than not that means no EJB, just JPA. Even more often it means just a servlet container instead of a full JEE server. I never understood the point of trying to use all that's available just because it's available; that is one sure route to overly complex systems.