Effective DevOps - how should programmer act under strict organization structure

[holly wang]

we have very strict rules in our organization. Application owner and programmers are not allow to work on any production servers even given our servers are all Linux servers which allow multi level users to log in at the same time. application owners have to go through system operation team for any minor change with their applications. did you book has any coverage on this topic?

[Junilu Lacar]

This, IMO, is more indicative of a larger and more difficult issue: that of culture, both on the management and the engineering sides. As many proponents of DevOps may tell you, the technology changes are not as difficult as the culture changes. I think this is true for most of the other questions that have been posted so far. I'd be interested to hear from our guest authors though.

[Tim Holloway]

I worked in that sort of an environment more than once. Even in shops where I can touch production systems, I try to make sure that I'm set up not to. If I don't have security rights, then it's harder to blame me if it breaks.

The processes I worked with there typically involved local desktop work, a beta test machine, and production. We developed locally, deployed to the Beta machines for independent testing - and sometimes users testing to see if they like what they asked for - then bundled it all up along with change control documentation and handed it off to Operations.

The primary difference between today and how I did it back in my mainframe days is that now we have a plethora of provisioning tools. Stuff like Puppet, Chef, Ansible, Salt, and so on and so on. Along with the integrated build systems such as Jenkins and all the other cool toys designed to help manage complex systems. Back in the day, not only did we have to walk uphill in the snow, the only deployment aids we had were all home-grown (I worked on a few - I was doing DevOps long before the term had been coined).

If you have a build system such as Jenkins, you can set it up to automatically build and deploy-to-test when you commit source code changes. If the tests pass, then the actual deployment team has a set of nice neat little packages that they should be able to plug into deployment modules.

Mind you, I wasn't starting from scratch. Even before I got the current generation of goodies, I was building WARs that configured from outside (so that the exact same WAR ran on desk, beta and production machines). And building OS packages instead of just dumping loose WAR files, scripts and executables on the operations staff.

[Jennifer Davis]

Holly,

Thanks for sharing a bit about your environment.

I think this is one of my favorite questions yet due to the participation of other folks in the thread! Love to see this kind of interaction.

So, Holly, what you've described sets off a few alarms around cultural challenges in the environment as Junilu rightly pointed out. It is completely ok to have an environment that due to compliance issues doesn't allow individuals to commit change to the live systems. Ideally this is true for the system administrators as well from a manual change stand point. All changes should be in code where there is transparency of folks to be able to see the changes and replicate them, and then automatically deployed to the live systems.

What you are describing sounds a little different from that, in that all change must be okayed by system operations team prior to the change happening. This sounds like a bit of fear driven deployment where the fear of change is associated with a large amount of risk which impedes everyone from doing their job.

Our book does indeed cover this topic throughout the book, as it's part of the whole larger than technical challenges scope that many organizations face. Specific technology will arrise and be better for specific conditions in an environment. Many folks focus on the outcomes that other organizations experience with specific technology. Some organizations look at the changes happening and ascribe the value of change "only being possible in those kinds of environments because in our environment we have these tools and technology that we must use". This is a big misconception that is leading to a lot of disruption in different areas as newer companies are able to displace larger companies due to their ability to rapidly change.

So some first steps in your organization to look at:

1) Do your development and test environments look like production so that when you hand off change to the system operation team they can feel confident enough in the risk of said changes?
2) Is there open communication between teams or does it get filtered through specific individuals with a lot of processes?
3) Is the reasons for why change is done the way that it is understood? If not, is it possible to have open discussions with people about it? (This is a sign of a problem in the environment if folks just respond 'because that's the way that it's always been done' or 'because it's the best practice')

I look forward to more threads like this one where folks chime in with their experiences as well! Thank you Junilu and Tim for your contributions!