Migrating our production environment from debian to OpenSolaris I wanted to simplify our configuration management recipes along the way. What I came up with is a mixture of Puppet style manifests and Capistrano backed ease of use in a new open source project called: Carpet.
Archives for January 2009
Last week I made an honest mistake that caused severe downtime on our website. Of course, I spent a lot of time thinking about all the contributing factors which ultimately caused such a nexus in the space-time continuum to appear over my keyboard at that crucial moment when I hit the enter key. It didn’t take long to realize what I had done – within a few minutes I heard the first murmurs of “WTF?!” coming from my colleagues.
It turns out that one of the factors which caused the critical failure had been implemented a good year before I had even started working for the company. And of course, there were multiple people involved and of course all of their implied, silent assumptions about how others worked. But regardless of all these “setups for failure”, at the end I was the one that did it – I was ultimately the one responsible for pulling the trigger which obliterated our site within minutes.
This is a guest post by Andrew Shafer, who is part of Reductive Labs, the people behind Puppet. Reductive Labs is helping people build better systems with better tools and processes. Andrew has been on several Agile software teams in various capacities for the past few years, and has a passion for applying Agile principles to operations processes.
First I’d like to thank Matthias and Dan for their insights on Agile Web Operations and for giving me the opportunity to post as a guest.
Choosing Your Weapon
Puppet or Capistrano appears to be a reoccurring question for many and I hope my opinion might shed practical light on the topic. In my mind, the question is a bit like asking a carpenter if one should use a hammer or a saw. Use an appropriate tool for the job at hand.
After using Pivotal Tracker myself for a couple of weeks, I recently migrated our complete development from Mingle to Tracker.
Migrating existing data between tools using CSV is always a pain. It starts with Mingle using tabs as a separator instead of, well, commas. Of course, you have different field names to match up, and then there’s the difficulties of making MS Excel write a CSV file with commas as separators instead of semi-colons. To avoid all that, I put together a super simple Ruby script to convert a Mingle export into a valid Tracker import:
Have you ever taken a midnight drive down a dirt road without any headlights on? While its certainly a thrilling (and stupid) thing to do, I certainly wouldn’t recommend doing the same thing with your data center. Do you have any idea if the load your servers experienced this morning was unusually high? Could you tell me how many MySQL “slow queries” were executed on your database yesterday? Have customers and co-workers praised you for your website’s performance?
Like Dan did recently, I was setting up a Subversion repository one year ago. Of course, I also wanted to have regular backups of my Subversion repository. As I was tired of writing bash scripts for such a task, I looked into writing a simple Ruby script for backing up my Subversion repositories.
Full and Incremental Backups
So most companies shoot middle of the road – granting access to a select group of individuals and ensuring there’s some decent logging to monitor usage and proper sick leave and vacation coverage. The question I’d like to help you answer is which individuals should have access. And, of course, like any sane individual, I’ll start my answer with ‘It depends…’ – it depends on the degree of configuration management your environment has.
Last week, I showed you how to setup a secure subversion server. Today, I’d like to show you how to technically accomplish a couple of development themes that are near and dear to my heart. The first is creating a quality gate with a release branch and creating a team of committers who are the only developers allowed to write there. The second is increasing communications between offshore development and inshore business which we’ll tackle with enforcing svn commit messages and svn diff email notifications. To accomplish this, I’ll introduce you to the basic subversion commit hooks: pre-commit and post-commit.
Fine Grained Subversion Permissions Using pre-commit
We want to ensure that not everyone has free roam across the entire repository. For instance, say we only want committers to have write access to the trunk and developers should be able to create branches whenever they want. Let’s use subversion’s pre-commit template to enforce some basic read-write permissions.