This week's giveaway is in the JDBC forum.
We're giving away four copies of Java Database Connections & Transactions (e-book only) and have Marco Behler on-line!
See this thread for details.
Win a copy of Java Database Connections & Transactions (e-book only) this week in the JDBC forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Devaka Cooray
  • Knute Snortum
  • Paul Clapham
  • Tim Cooke
Sheriffs:
  • Liutauras Vilda
  • Jeanne Boyarsky
  • Bear Bibeault
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Ron McLeod
  • Piet Souris
  • Frits Walraven
Bartenders:
  • Ganesh Patekar
  • Tim Holloway
  • salvin francis

How to manage external configuration values?  RSS feed

 
Marshal
Posts: 4630
300
Clojure IntelliJ IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

I'm working with a system that currently has its configuration stored in the code repo and that is packaged up into the build artefact. There are many reasons why this is not ideal and I am working to extract them from the build and define them externally, most likely through environment variables but maybe some external config file.

My question is how do I manage external configuration? Do you keep them in separate repos? A Wiki page? Other?

Thanks, Tim
 
Sheriff
Posts: 13480
222
Android Debian Eclipse IDE IntelliJ IDE Java Linux Mac Spring Ubuntu
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Tim Cooke wrote:... configuration stored in the code repo and that is packaged up into the build artefact. There are many reasons why this is not ideal


Let's unpack that a little bit. What are some of those reasons?

I ask because my experience is that having environment settings versioned and committed to a repo is in fact a good practice.

A few reasons:
1. If you always pair the various environment settings that apply to a particular build artifact, you'll minimize "It worked on my local machine" problems. Maintenance becomes easier and less error prone.
2. You can automate environment setup and make it part of your deployment pipeline
3. You can deploy the same artifact to different environments; you don't have to do separate builds for each separate environment. This seems like common sense but I have had clients who didn't do this and it was chaos all the time.

See:
https://docs.chef.io/environments.html
https://docs.chef.io/cookbook_versioning.html

http://www.glennposton.com/posts/puppet_best_practices__environment_specific_configs
 
Junilu Lacar
Sheriff
Posts: 13480
222
Android Debian Eclipse IDE IntelliJ IDE Java Linux Mac Spring Ubuntu
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Tim Cooke wrote:A Wiki page?


The settings values themselves, absolutely not.

Where a wiki page can be of tremendous value though is when it documents the approach and details/intricacies of how to maintain various environment settings, what kind of collaboration/communication between team members and to/from entities outside the team needs to occur, links to relevant policies and brief descriptions of how those policies apply to the software, etc.

There's a lot of information that can be useful on a well-maintained and up-to-date wiki. Concrete examples that show what you'd set and how those values affect operation of the software are good, too, but certainly not the settings values themselves. My experience is that those can quickly get out of synch with reality. When they're in a repo and part of an automated deployment process, developers are incentivized to keep them up to date and synchronized with the build. On a wiki as static text, not so much.
 
Tim Cooke
Marshal
Posts: 4630
300
Clojure IntelliJ IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for the detailed reply Junilu, much appreciated.

The problem I have is that because the settings exist in the same repo as the codebase and gets packaged up in the jar then it couples the build to the environment. Let's say that the database user credentials change, oh shoot now we need to build, test, and deploy a new version of the application. This is sub optimal.

My thoughts are to externalise the configuration and maintain it separately, perhaps in a separate repository, so that the same build version can be deployed in different environments. So would it be a good idea to create a separate repository for configuration? Perhaps even separate repositories for Dev and Prod configurations to minimise the visibility of sensitive information such as the credentials to the live payment gateway?

Thoughts?
 
Junilu Lacar
Sheriff
Posts: 13480
222
Android Debian Eclipse IDE IntelliJ IDE Java Linux Mac Spring Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Tim Cooke wrote:Let's say that the database user credentials change, oh shoot now we need to build, test, and deploy a new version of the application. This is sub optimal.


Would changing the credentials obviate the need to test the application if they were stored in a separate repository? I would think you'd have to do at least a smoke test either way. So it's only the build and deploy processes that would be somewhat of a bother but then if those are automated, then the waiting for those process to be done is really your only biggest annoyance.

DB credentials are kind of special though and often require different approach to management from other environment settings. See secrets management strategies.

I've seen both approaches used where environment configuration is separately managed from the main build/deployment artifact. The downside to this as I see it is that there's some effort required to make sure you manage dependencies between the two properly. Like say rev.150 of environment configuration settings only applies to build.2511 or later or only from build.2511 to build.2532.  I imagine there should be a way to automate or semi-automate mapping between paired versions though so wouldn't be that much effort. You just need to make sure there's an easy way to verify that pairing and ensure the two sides are in synch and properly matched in any given deployment environment.

In my past life as a tech lead, we were deploying our apps to shared enterprise infrastructure that used deployment scripts that were not under our control. This required us to follow a standard convention for packaging up our app and that meant we had to keep environment configurations together with our application code. Basically, we had to have configuration settings for all possible deployment environment included in the EAR file like this:

EAR
:
... /deploy-config
    |.. /dev
    |    |.. config.properties
    |
    |.. /stg
    |    |.. config.properties
    |
    |.. /load
    |    |.. config.properties
    |
    |.. /prd
         |.. config.properties

The deployment scripts would unzip the appropriate config.properties from the EAR file based on a preset environment variable, e.g. LIFECYCLE, where the values were "prd" in the production instances, "dev" in the development instances, and so on.
 
Junilu Lacar
Sheriff
Posts: 13480
222
Android Debian Eclipse IDE IntelliJ IDE Java Linux Mac Spring Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Always consider context though. The example I gave above where we had to maintain a config file for each environment was manageable because we had a very limited number of environments. If you're luckier and can create multiple deployment environments, like if you deploy to a virtualized environment where you can easily spin up a server instance, that changes. See twelve factor app section 3, "Config" which discusses the same issues.

In section V, "Build, Release, Run":

“Code becomes a build, which is combined with config to create a release.”

From: Wiggins, Adam. “The Twelve-Factor App.”

 
Tim Cooke
Marshal
Posts: 4630
300
Clojure IntelliJ IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I'm dipping in and out of thinking about a solution to this problem, specifically the solution I'm going to apply to my system, so apologies for the long periods of radio silence. As always Junilu I appreciate your input very much.

I was aware of the 12 factor app and have now read over the config and build sections many times. I'm currently focussed on this statement:

12factor.net wrote:The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.


I completely get the idea here of using env vars and it sounds very appealing to me but I don't have a good idea of where to store the configuration values. The app is Spring Boot so can have a single application.properties file where the values are placeholders for environment variables

application.properties
That gets me a single configuration file within the repo that defines the configuration items that are required. This will be an improvement as currently we have multiple config files for different environments and some extra that devs have added for their own testing and one of the problems is that the set of config keys are not consistent across these files where new items have been added to a dev config file but not the prod one.

If I go with this approach then I have the configuration items in my repo with no values, which is good. But now I'm back to the same question of where do I keep my config values?
 
Eliminate 95% of the weeds in your lawn by mowing 3 inches or higher. Then plant tiny ads:
how do I do my own kindle-like thing - without amazon
https://coderanch.com/t/711421/engineering/kindle-amazon
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!