danja.
How hard could it be? Well, like with most engineering endeavors, it’s about 99% preparation and 1% implementation. Unlike Continuous Integration, however, continuous deployment (CD) involves business as well as technical considerations. Let’s outline them and see how close (or far) we are to the holy grail of DevOps.
Continuous Deployment is founded upon Continuous Integration
CD takes your CI process to it’s ultimate conclusion. After a successful build and automated tests, the changes are automatically pushed to production. Your CD process will only be as good as your CI process.
A good CI means you’ve got your technical ducks in a row. Sufficient test coverage combined with test environments that could easily be used in production in an emergency are indicative of considerable engineering effort on both the development and sysadmin teams.
My use of “sufficient” is no accident. I’m not going to give some magic number here, because it depends on your business. I know our current code coverage of 35% is enough for us to avoid full regression tests before making releases (which we do many times a week).
Business at the Speed of Thought
Not only was this an excellent book written by Bill Gates in 1999, but continuous deployment is the process that enables it to happen. The problem is that most businesses (like a lot of people) tend to speak before they think.
“Why is ‘widget A’ already live? I didn’t clear that for release.”
“We planned that story together for this sprint and I finished it this morning.”
“Wow – but, I didn’t clear this idea with (marketing|management|shareholders)…you have to disable it ASAP!”
This has happened to me more times than I’d care to count. Unfortunately, folks are so trained in waterfall-style thinking, they usually reserve resources for their idea months in advance in the hopes that it might be implemented before the deadline.
Until your product owners are used to your CD’s quick turn-around times, use configuration flags for enabling and disabling of “risky” features (i.e. new or majorly refactored advertisement placements). I recommend some “alphaConfiguration” file to store these temporary flags. You’ll be able to keep a better overview of all the alpha features and, of course, can remove them once they’ve finally been approved for release.
These flags give you the all-important integration step in a tightly controlled fashion. Individual commits are small changes, and the sooner you get these changes live, the sooner you detect problems. This lean methodology
Continuous Deployment is the Holy Grail of DevOps
Isn’t this a bit over the top? No, not at all. CD means that developers are making production changes with every commit (albeit via build server proxy). Operations will have 100% confidence in their infrastructure including all the necessary monitoring and logging before they hand over this amount of control.
There are countless success stories out there: Etsy, Heyo, IMVU & Atlassian all do CD.
How close am I to CD? I’m really only missing the “alphaConfiguration” file to effectively hide half-baked ideas from public consumption. How about you? What are your major obstacles to deploying continuously to your live servers?
Agile Web Operations 
There are obstacles to deploying meaningful changes continuously to live production servers. Like making interface changes and certifying them with partners, like making database and data changes, like security checks. Most of the changes that people talk about releasing continuously are trivial: small changes or small fixes. Anything bigger has to be done more carefully. Schema changes can’t be made continuously. Bigger changes can’t and shouldn’t be made continuously, even with dark launching. Etsy doesn’t continuously deploy public-facing changes, they take their time and review them and think them through like any sane organization. See John Allspaw’s talk at Surge last year.
Automated testing isn’t enough for every (any?) system or every company, certainly not automated testing at 35% coverage which is much lower than say Jez Humble recommends in Continuous Delivery. And you have to account for code reviews and security checks, which could be done before check-in – which is how some companies are able to achieve Continuous Deployment (they move some of the necessary steps like code reviews up before check-in, so you know the code is already pretty good). Too many descriptions of Continuous Deployment make it sound too simple and too easy. It’s not, and even organizations that have a lot of experience with it continue to have security and reliability problems (Facebook, WordPress, ….). problems.
Yes there’s a lot to learn from Continuous Deployment, about streamlining and simplifying release and deployment, and reducing risk by breaking work down into smaller and smaller pieces and tieing all of this together with ops monitoring and metrics. But it’s not the “Holy Grail of Devops” or at least it shouldn’t be.
LikeLike