Good way to manage customized versions of the same base application

[Marco Ehrentreich]

Hi folks!

Sorry if this question doesn't perfectly fit into this forum but it seemed to be the most appropriate place because it's a lot about application design...

I need some advices what would be a good way to manage different, customized versions of the same application plattform which are in large parts identical but have to be adapted to different customer's needs. The application I'm talking about is a Java web application and the problematic parts are mostly XML templates for the (self-made) view technology and XML files for message (similar to Java's bundle files). The codebase is basically the same for all customers.

Additionally it's not only possible to have customized versions of this application but the customized versions also reuse templates and message files from the base application which can each be overridden in the custom application as needed. In general this sounds like a good idea but it isn't. The problem is that the templates also support includes, too much code logic and often it's unclear if specific changes have to be made in the base application (for maximum reuse between customers) or if they should be made in the customer version itself to override the base application templates. Over time this whole thing has become more and more confusing because many people made changes and so there's a lot duplication because often you can hardly tell if some template parts or messages might already be implemented somewhere else.

I hope it's more or less understandable for you what's the problem. Does anyone have experience or good ideas how to manage such concurrent versions of a similiar base application which allows for maximum reuse with minimum duplication and is still easily understandable (as it would be nice if someone with less experience could make smaller changes to templates etc.)?

Thanks in advance!

Marco

[Ilja Preuss]

Frankly, it sounds to me like you are having more of a process problem than a design problem. It's hard to tell from the distance, but I would expect your main problems to include

- you are accepting mediocre code quality (most importantly duplication), and
- your developers aren't communicating enough

Are your people under a lot of time pressure? This often happens when people feel that they are rewarded for doing things the quick and dirty way - which, in the long run, of course leads to maintenance hell.

Does that sound like it's on the right track?

[Marco Ehrentreich]

Hi Ilja,

it doesn't only sound like it's on the right track, it IS exactly on the right track!

I think it's the well-known result of a fast-growing development team and application without enough time or care to adapt the processes fast enough as well. Unfortunately the template and message management system is all self-made and the code suffers from the same problems. So I'm not even sure if this can be fixed in the existing application with reasonable effort. I'm more or less doing some research for future developments to prevent such problems.

Anyway, as you might have guessed the maintenance hell is already a fact and so I'd be glad to hear some suggestions for a technical solution which helps to actively force better habits when editing templates etc. It turns out that the current solution is doing to many hidden things behind the scenes because the merge and override process for template and message files happens when the application is packaged or even at runtime. So the end result is very confusing and unpredictable for someone without a deeper understanding of what the code or build system does internally. And of course it would be nice to let other people like web designers or translators edit these files without having to know all the details.

I'd already thought of something like the patch system in UNIX so that it is at least possible to produce the merged end results for templates and messages at development time by "patching" the base application. I think this solution would still be far from ideal but at the moment I can't see any really good solutions to this mess. On the other hand I'm pretty sure that a lot of companies will have the same problems when an application has to be customized AND maintained in some way for different customers without unnecessary duplication of data and maintenance work.

Marco

[Ilja Preuss]

I'm sorry, I don't know of a good solution for your design problem. But it sounds as if you already have an idea - and I'm sure that if you encourage your colleagues to tackle the problem, you together will come up with even better ideas.

In my experience, it's almost possible to fix an existing legacy system, and almost always also less risky and more viable. And that's because it can be done in small, iterative steps.

There are three main ingredients you need to do that:

- it really being a top priority, acknowledged by your management, (or the willingness of your team to work as professionals against the will of management),
- the resources to do it, and
- the skills to do it - mainly refactoring skills, to implement the incremental changes in low risk ways, and the skills to write automated tests for legacy code, to support the refactoring

For the latter, I highly recommend the books "Refactoring" and "Working Effectively With Legacy Code". I also highly recommend getting a coach, or at least some training...

[Marco Ehrentreich]

I'm sorry, I don't know of a good solution for your design problem. But it sounds as if you already have an idea - and I'm sure that if you encourage your colleagues to tackle the problem, you together will come up with even better ideas.

Hopefully we can come up with a better idea. But as I said I'm doing just some research up front, so chances are good we will find a solution.

In my experience, it's almost possible to fix an existing legacy system, and almost always also less risky and more viable. And that's because it can be done in small, iterative steps.

Unfortunately I don't have very much experience with legacy systems, so it's hard to tell if it would be successful enough. I'll do my best and take care that everyone including me is well prepared to give it at least a try and fix as much problems as possible.

- it really being a top priority, acknowledged by your management, (or the willingness of your team to work as professionals against the will of management),
- the resources to do it

This is really hard to convince management to invest some time and resources to improve an application that is obviously working - at least on the surface ;-) Luckily I was able to give some insight in the problems that already are and will come in the near future, so we have some resources and can officially work on improvements.

- the skills to do it - mainly refactoring skills, to implement the incremental changes in low risk ways, and the skills to write automated tests for legacy code, to support the refactoring

You can read everywhere that it's absolutely necessary to have a basic test suite before going to fix a legacy application. As I said I don't have enough experience with legacy systems, so I can't tell. But just to be sure, would you dare to refactor a hard to understand legacy system without tests? I personally can clearly see the value of unit tests for such a project but that's a point which will need some more convincing arguments why it's worth the effort to create unit tests for an existing application...

I highly recommend the books "Refactoring" and "Working Effectively With Legacy Code"

Of course "Refactoring" is a must have for every developer ;-) I recently started to read Michael's excellent book, too, which has indeed lots of valuable advices for refactoring legacy applications. I'm not yet finished with it but I'm sure there are many things we can use in practice to improve the quality of our application.

Thanks a lot for your thought's about these problems!

Marco

[Hong Anderson]

Have you tried to use Subversion branching feature?

[Marco Ehrentreich]

Hi Kengkaj,

thanks for your answer!

In fact we're using Subversion for configuration management and the said application not only exists and is maintained for different customers but also in different versions but all customized versions of the same release version are located in the same branch.

Would you recommend to branch off each customized application, too? I think so... It sounds like an interesting point but I haven't yet tried extensively enough if it would work well for the XML templates and message files as explained above. I guess an important point for success would be a really good diff tool which allows easy merging of XML files. Or would you recommend a completely different approach?

Marco

[Hong Anderson]

Actually, Subversion branching feature was especially designed to satisfy your requirement.
You can have trunk which is main development line, if you have customize versions that are slightly different you can create branches, after than when trunk has changed you can merge changes from trunk to branches.

About XML, message files, I don't think there will be a problem because they are text files. Merging is not easy anyway, but you should try it first, for me built-in merging function of TortoiseSVN is good enough.

[Marco Ehrentreich]

Actually, Subversion branching feature was especially designed to satisfy your requirement.

Of course you're right and as I said we're already using Subversion for release managment and development branches of our codebase, too. It's just the XML templates which are duplicated "by hand".

About XML, message files, I don't think there will be a problem because they are text files. Merging is not easy anyway, but you should try it first, for me built-in merging function of TortoiseSVN is good enough.

I guess this is more a problem of our process, as Ilja pointed out. At the moment the XML files are simply to chaotic because updates in the past were not made in a well-defined way and there was no use of version control at all. But I should definitely give it a try with some cleaned up test files...

Thanks a lot for this advice!

Marco

[Ilja Preuss]

I would never, ever use version control for the management of customized versions.

Merging is already laborsome enough if you just use it for versioning, where the number of active branches is small, life-time of branches relatively short and the direction of merging definite. Merging between (potentially dozens of) long-lived versions that don't have a definite hierarchy sounds like a maintenance nightmare to me.

[Marco Ehrentreich]

Now that you say it, this is exactly the impression I have at the moment!

It's already hard enough to keep different long-lived release branches which share a common base. That was the reason I had never thought about the idea to put the customized versions under source control, too. And I'm pretty sure it won't work without a clearly defined way where and how to put in which changes.

Other than this I still couldn't come up with a really good and maintenable solution but maybe it still takes some more time


Marco

[Hong Anderson]

Whatever solution, it's not necessary that it must be easy for maintenance. Something, even in its simplest form is still not simple. That applies to everything not just managing customize versions .

[Marco Ehrentreich]

I know that it's usually hard or impossible to find a solution for a problem which is much simpler than the problem itself. As it's hard to find a good solution for our problem, I guess the problem is more complex than it seems at first and so the solution won't be too trivial.

But I'd disagree regarding maintenace. I asked for help because we're already experiencing maintenance problems day by day which most probably will grow even more over time. I think a key difference in comparison to "normal" development artifacts is, that we're talking about view templates and I18N message files. It would doubtlessly be nice to have some experts for designing the views or translating message files into different languages. But because the system is so confusing and hardly maintenable these tasks can currently only be done by developers. I guess this is not the situation you'd prefer

Marco

[Hong Anderson]

Well, I mean if we have difficulty at 100% and we can reduce it to 80% (maybe it's the easiest solution), it's still not easy but it's better, isn't it?

If anybody has solution that is easier than Subversion branching, please suggest, I want to learn too.

[Marco Ehrentreich]

Of course 80% is better than 100% of all problems. But you will surely agree that if any solution will mean quite some effort, we'd prefer to invest some more work and time to find a solution which is hopefully <50% But seriously, I believe that even with near 50% this whole thing would still be a big mess. So it's questionable if it's worth all the effort, if we can't come up with a solution that works really well and solves most of the problems.

Anyway, I'd still love to hear some other suggestions, too! Are our requirements really that unusual? I can hardly believe this... I guess there should be many companies which provide a customized version of an application for different customers.

Marco

[Ilja Preuss]

I'm actually working on a project that is highly customizable, in many different ways. We are using a combined approach:

- typical smaller customizations are done via pure (XML) configuration of the main platform
- medium sized customizations are done via injection points for Groovy scripts
- strong customizations are done via customer-specific plugins (which are their own svn modules)

Perhaps that gives you some ideas...

[Marco Ehrentreich]

Hi Ilja,

thanks for your advices. This sounds like a reasonable mix to me. In particular your approach with Groovy scripts as plugins is very interesting. I read about it some time ago and would like to give a try if as soon as I see any opportunity. Would you recommend this way of externalizing an interface for plugins after having some experience with it?

After thinking about our problem some more, I decided that the real problem comes down to the question how a concept like inheritance in OOP can be implemented for XML files in a clean and maintainable way. After all the said XML files are the only troublemakers which are too hard to handle. I'm going to ask for help regarding this idea in the XML forum.

Anyway, thanks a lot to both of you for taking the time to discuss my problem

Marco

[Jimmy Clark]

... I guess there should be many companies which provide a customized version of an application for different customers.

Good guess. Customization is typically driven by configuration data stored in a relational database. This is the best route...as always, data is best stored and accessed via relational database management systems (RDBMS).

Keep in mind that these system required clean, performant, and well-written code which has been adequately tested for both technical and quality-of-service requirements. And there usually is a significant portion of system code which executes inside the RDBMS (not a Java JRE).

[Hong Anderson]

I read this topic again, it seems that Marco has already used different XML templates for different versions.
I don't know how the application selects XML files, but I think the problem is not configuration but it's unclear design/implementation.

Do I still misunderstand your problem? But if I understand correctly, could you please to post a code example and explain what/where the problem is?

[Marco Ehrentreich]

Hi,

thank you all for showing still some interest in this problem

@James: You're surely right that a central database would be a good place to store application wide data. Unfortunately I think this would become quickly impractical in this case because we're talking about XML based view templates which often have to be edited by hand. Of course it would be possible to provide a small tool to upload and download the templates to and from the database but I think this wouldn't change much of the problem as it only shifts the problem from the file system to the RDMBS

@Kengkaj: After taking some more time again to think about my problem, I came to the same conclusion regarding bad design. Unfortunately it's hard to construct a very short example to show the problem but I'll try to explain you my newer insights. First, the XML message files are not that big deal, so we don't need to discuss about that any longer.

The XML based templates for the views of this (web) application are the real mess for several reasons:

1) The templates exist in the base application and have to be customized depending on the requirements of a customer. At the moment this is mainly done by duplicating the base version and copy it into the customized application because each XML file in the customer application can only override entire file of the base application and ugly duplication by copying a file is the fastest (but less maintainable) way to this.

2) There's no 1:1 mapping between XML templates and web pages. The XML templates are rendered by a self-made template engine which allows, in my opinion, way too much code-like elements. For that reason one template often contains the content of many web pages which are rendered based on some conditional logic in the templates themselves.

3) Additionally the templates may include other template fragments. Even this is often done based on some conditional logic. To make the problem worse every customized application has full access to all templates of the base application, so you often can hardly figure out from where some other fragments are included.

4) Another main problem is, that the templates contain too much code, i.e. not only the basic template engine features like loops for iterations or conditional decision logic. No, these templates may contain real method calls which are pulled out and executed via reflection at runtime.

5) Nearly all of the things above happen at runtime. You simply can't see or test the full templates without the running web application. This is very undesireable and you can imagine that this is nothing you can expect non-technical people to edit and maintain.

6) One last big problem is, that you're not done with the templates. There is a mix of structural and design element all over the place. To produce the final output you can see on a web page there's quite a bunch of things the template engine has to do and additionally there's one more step in this process which involves a XSL transformation. Not to speak of CSS files to tweak the design details depending on the customer.

That said I came to the conclusion that our problem is partly technical (the code and reflection things), but a lot of the problems come from misusing the template engine and missing clear definitions where to put specific content. I guess it doesn't make much sense to spend more of your time thinking about this. There's a lot we have to do ourselves to clean up the mess.

Anyway, this discussion has helped me a lot to identify the real problems ;-) Thanks again for taking the time!

Marco

[Ilja Preuss]

James Clark wrote:Customization is typically driven by configuration data stored in a relational database. This is the best route...as always, data is best stored and accessed via relational database management systems (RDBMS).

Uh - why?

[Ilja Preuss]

Marco,

do I understand correctly that those templates that are customized in fact also contain application logic that should *not* be changed, and probably even need to be changed in every single template whenthe application itself changes?

With other words, it sounds like you possibly don't really have a physical separation of things that are supposed to be customized, and things that aren't?

[Marco Ehrentreich]

Hi Ilja,

you got it exactly! Can you understand why I said, we already have arrived at the maintenance hell central station?

I'm relatively new in this project/company, so I can't really tell you why or how the problem could grow to such a big mess. But I was able to convince my colleagues and management that something has to be done soon. As you can imagine the templates are not the only problem.

You don't have to tell me how bad this situation is. Sometimes it really takes hours to days(!!!) to modify the necessary parts for only one single web page. As I said, I'm new to this project but this is surely not the main reason that every change takes ages

During our discussion here my thoughts about the root cause for these problems have changed a few times and as I wrote in my recent post, I've come to the conclusion that we have far bigger problems than the technical ones. Unfortunately to improve the situation the time pressure you already talked about comes into play. But even with the full permission of management the human resources on the team won't be enough to fix all major problems, I guess. I'm unsure how to get all things done but I hope it's possible to find a way to break this vicious circle...

Marco

[Ilja Preuss]

Sounds like you are starting the journey of a change agent. Good first steps.

You might be interested in the book "Fearless Change" - it has lots of advice on how to lead change, even if you are not in charge.

Enjoy the ride!

[Marco Ehrentreich]

Thanks a lot

I hope this journey doesn't end short, but at least at the moment everyone seems to be motivated to turn for the better.

The book is already ordered right this moment It sounds very interesting although I can hardly believe that patterns can be applied easily to a topic like change management. Obviously the reviews of the book tell that they can!


Marco

[Jimmy Clark]

Uh - why?

Because attempting to do otherwise typically creates a variety of maintenance and versioning nightmares.

[Jimmy Clark]

@James: You're surely right that a central database would be a good place to store application wide data. Unfortunately I think this would become quickly impractical in this case because we're talking about XML based view templates which often have to be edited by hand. Of course it would be possible to provide a small tool to upload and download the templates to and from the database but I think this wouldn't change much of the problem as it only shifts the problem from the file system to the RDMBS

The text of the XML-based view templates and the meta-data about "how/when/where/who" about these templates are not the same. The problem you have been describing is a data management problem, not necessarily an "XML" problem. A commericial grade RDBMS has the tools readily available for you to create a more efficient way to manage the templates.

[Marco Ehrentreich]

Hi James,

I think now I can see how a RDBMS would fit in this problem. I'm just not sure how to realize a solution. If I understand your last post correctly, you'd propose to shift the problem which part of templates belong to which customer and are included under which circumstances etc. to the database. Did I get this right? This seems like a plausible technical solution.

As you may have read in this thread, at first I have to solve the non-technical issues before thinking about a better implementation. That means convincing everybody to improve the system and doing some clean up on the existing mess. But I'd definitely like to hear your ideas!

Marco

[Jimmy Clark]

Yes. Each customer's "configuration data", i.e. meta-data about what they see, what is displayed, what they can do, what they can't do, security settings, etc. is stored in a database. The application is guided by this meta-data, and also default meta-data.

[Marco Ehrentreich]

OK, now I see what you mean!

Aside from one or two configuration files all the so-called meta-data or configuration data are indeed stored in our RDBMS. But this is not really the problem. In fact it doesn't even influence the view templates directly.

The behavior in the view templates is the problem: only parts of a template visible depending on a condition, conditional inclusion of other templates, (partial) overriding of templates from the base application, "code" which class real Java code with reflecteion and other nice tricks... And all this is not driven by some data from the database or something like that, but mainly by the application flow at runtime! So in this part of the big picture I don't see how a database would really fit in except for storing all XML templates in it which wouldn't make the system more maintainable, in particular not more manually maintainable. This is really a design problem with the entire system, I think.

Marco

[Jimmy Clark]

Aside from one or two configuration files all the so-called meta-data or configuration data are indeed stored in our RDBMS. But this is not really the problem. In fact it doesn't even influence the view templates directly.

This is apparent. Currently, you may not be storing "any" meta-data about the view templates. The solution to your problem does not exist yet

The "behavior" of your view templates can be controlled with meta-data. Different customers can experience different behavior.

And all this is not driven by some data from the database or something like that, but mainly by the application flow at runtime!

This describes what exists today.

In regards to your view templates, the meta-data in the database would be a fundamental part of the application flow at runtime!

You seem to be getting it, slowly... keep thinking

[Ilja Preuss]

James Clark wrote:

Uh - why?

Because attempting to do otherwise typically creates a variety of maintenance and versioning nightmares.

Oh, I see - I misunderstood what you were talking about.

We simply use a folder structure for customers, and SVN for versioning. Seems to work perfectly well.

In fact, I'd think that using anything else but the same versioning system as that for the code would lead to, if not maintenance nightmares, at least unnecessary maintenance overhead. But perhaps I'm missing something...

[Marco Ehrentreich]

Unfortunately I still didn't get i, James... I mean, basically I understand what you're talking about but I suppose you should see those "templates" in action to understand my point. They are not only ordinary templates you probably know from other view technologies. Some of these beasts are alive That is they have behavior almost like a piece of code thanks to the magic going on behind the scenes with reflection and the like. Of course this should have been avoided in the past and could be cleaned up with quite a bunch of work which in turn would reduce a lot of the maintenance burden.

But even if the templates would be well-structured and all but the essential template logic would be removed I couldn't still see the real advantages of putting templates into a database. Of course you're right, data should be kept in a database but in my opinion such templates are not the data I'd want to have all in a database. If they are regular files managed with a typical versioning tool I can tell any person to simply check out the template directories and edit the files as any other file on the computer without having to know how to handle databases. In contrast if the templates are stored in our RDBMS I'd need at least some kind of tool to make it as easy as possible for web designers etc. to access and manipulate such files. I don't believe those people would be amused if they had to take care of database stuff.

Perhaps you could be a little bit more precise regarding the "meta-data" you're talking about. Maybe I'm still missing something to fully understand the solution you've suggested.

Marco

[Jimmy Clark]

Unfortunately I still didn't get i, James... I mean, basically I understand what you're talking about but I suppose you should see those "templates" in action to understand my point. They are not only ordinary templates you probably know from other view technologies. Some of these beasts are alive That is they have behavior almost like a piece of code thanks to the magic going on behind the scenes with reflection and the like.

Sounds like there is a significant group of design flaws in this "magical" code and all the "wonderful" things that are happening. XML design with a narrow perspective usually results in XML that is difficult to manage.

I never mentioned putting the templates into database tables. I also have not mentioned a "solution" either. What I have conveyed is a way or path to a solution that you would have to create, if you can see it.

Information about the behavior related to how the XML structures are formed is what can be stored in database tables. This is how you can achieve efficient customization for each customer with the same application code. You really need to understand what needs to be done and would really need to redesign your current system.

Think about the current conditional logic, parameters, arguments, etc. that currently exist. Then think about how this varies from customer to customer. What is it about "customer" that makes this happen this way or that way?

You would pull this information out of the current code, but still use it in the current code

The data in the database can be transfomed into objects similar to how a struts-config.xml file is transformed into objects when a web application is deployed. Use these objects for guiding the behavior of the templates instead of the hard-coded stuff.

There will be a charge for any further analysis ... just kidding

[Ilja Preuss]

Marco Ehrentreich wrote:Of course you're right, data should be kept in a database

*Everything* is data. Even program code.

How data should be stored depends on a lot of factors: the structure of the data and in which ways it needs to be processed, just for example.

A RDBMS is a good place for data that maps well to the relational model, that needs to be queried in ways where the relational model is a good fit, and that needs to be accessed in transactional ways.

For other kind of data, there are alternative DB models, and non-DB alternatives. It would be quite ineffective to rule them out just because you are "dealing with data".

[Marco Ehrentreich]

*Everything* is data. Even program code.

OK, OK, I just said "data" but of course I only meant data which are appropriate to store into a RDBMS ;-) As I said I don't see any value in storing really _EVERY_ data into a database.

Sounds like there is a significant group of design flaws in this "magical" code and all the "wonderful" things that are happening. XML design with a narrow perspective usually results in XML that is difficult to manage.

You got it! But in fact it's not a significant group of design flaws. This whole thing IS a design flaw. On the sad days I often think to myself if it would be even possible to get everything into such a mess if you'd have to do it from scratch :-) I guess it would take a lot of time to come up with all the bad ideas and habits which have lead to this situation.

Information about the behavior related to how the XML structures are formed is what can be stored in database tables. This is how you can achieve efficient customization for each customer with the same application code. You really need to understand what needs to be done and would really need to redesign your current system.

Think about the current conditional logic, parameters, arguments, etc. that currently exist. Then think about how this varies from customer to customer. What is it about "customer" that makes this happen this way or that way?

You would pull this information out of the current code, but still use it in the current code.

We are simply talking about good application design, aren't we? Of course I don't have problems to imagine a well-designed application with some kind of database backend. I just couldn't see how this would help me in the current situation. Under "normal" circumstances it wouldn't have taken so long to convince me of your ideas ;-) Unfortunately it's just fact that a complete application redesign, which definitely would be necessary to even get near the situation you describe, is not worth the effort. As you can imagine the design of the whole application is quite similar to those templates. Therefore it would be impossible to quickly repair just the template part. At least there are future plans to start from scratch with the well-known requirements of the old application but hopefully without its problems.

There will be a charge for any further analysis ... just kidding

I don't get paid for the little research I'm doing here with you. This is all my FREE time so it's unfortunately FREE of charge But believe me, if you could see the creature in the shape of this web application or you even had to work with it, you'd be really, really happy to work 24x7 for free just to make any improvements

@Ilja: I was notified just this afternoon that the book you suggested has arrived at the local post office. I'm really looking forward to get some useful advices out of it ;-)

Anyway guys, the current situation doesn't allow me to do much more than a bit of clean-up work and it will take some time until a redesign of the whole application is in sight. I'll do my very best to do at least some improvements with the insights I've got from our discussion here. I really appreciate all of your advices and thank you very much for taking so much time

Marco