Puppet or Capistrano – Use the Right Tool for the Job

This is a guest post by Andrew Shafer, who is part of Reductive Labs, the people behind Puppet. Reductive Labs is helping people build better systems with better tools and processes. Andrew has been on several Agile software teams in various capacities for the past few years, and has a passion for applying Agile principles to operations processes.

First I’d like to thank Matthias and Dan for their insights on Agile Web Operations and for giving me the opportunity to post as a guest.

Choosing Your Weapon

Puppet or Capistrano appears to be a reoccurring question for many and I hope my opinion might shed practical light on the topic. In my mind, the question is a bit like asking a carpenter if one should use a hammer or a saw. Use an appropriate tool for the job at hand.

Basic Server Lifecycle

Before we dive into the tools, let’s pop up one level and talk about web operations to outline what we actually need to accomplish. At the most basic level, the server life cycle consists of provisioning, providing some services and sunsetting. Provisioning is taking a server from bare metal to setting up the services to be provided, and for now let’s just throw spinning up virtual machines in there too. Providing services is the point of the server’s existence and we will get back to that. While Sunsetting is the least interesting phase in some respects, I know you know of someone managing systems with a bunch of stuff running that may or may not add business value, but they are afraid to turn off the system because last time they did, the cron that delivers the report on the 3rd Tuesday for accounting didn’t run and all hell broke loose. Hopefully by the end of this, we’ll have a clear idea how this can be avoided.

A Destiny Of Service

Let’s skip the first part of provisioning and get to the point where we have a fresh OS installed. Not that this isn’t an interesting problem, but whether you are PXE booting or bringing up virtual machines, at some point you have a clean image. Let’s pretend we are smart enough that this is a fairly basic image and we haven’t gone down the path of image sprawl trying to version all the services that way. What should this server be doing? Should it be running a web server? Or a database? Maybe both? And some sort of middleware? Once we have answered these questions, how can we make it so?

The Road To Script Hell

The traditional sysadmin approach might accumulate a bundle of scripts with procedures to build the fresh system up to the running services. The question then becomes which script should be run in which order. So there are scripts calling scripts, with no real standard or convention and there are often organizationally specific details and logic embedded in the procedures. Furthermore, these procedures can often be destructive when run repeatedly. Maintaining and coordinating scripts often becomes problematic overhead which bleeds off more and more time. Maintaining scripts has no business value, running services do. At some level of scale and complexity, the overhead outweighs the benefit and this point probably comes more quickly than one might think. Trying to do this with Capistrano offers some advantages, but in the end the primary advantage is the expressive density of Ruby versus your favorite shell.

Fiat! Reframing The Problem

What Puppet provides is an abstraction framework for expressing services declaratively. This reframes the problem from ‘do’ this, then ‘do’ this, to ‘be’ this way. The overhead of maintaining detailed procedures that build a service step by step reduces to a semantic encoding of what the service should be. Puppet does all the work and the framework ensures the process of synchronizing managed resources is idempotent, or non-destructive. Once services like ‘web server’ and ‘database’ are generically described they can then be parameterized by the ‘facts’ (like fqdn, ip, OS, etc.) of a host and applied. The default Puppet configuration will resynchronize the services every 30 minutes and will restart services if their configurations change. Furthermore, since the systems are represented semantically with code and running that code is idempotent, changes to services can be propagated by updating their descriptions (in version control, of course). The resulting Puppetized infrastructure produces consistently defined services with a lowered cost of maintenance.

Capistrano And Puppet: Do Be Do Be Do

So where to use Capistrano? Puppet is good at putting things into a certain state. That state can be simple or complex, and can be migrated with the resynchronizations, but Puppet’s current model is not particularly suited for things than might need to be done right this second and coordinated across hosts. Which brings us back to a server providing services. Sometimes there is a need to ‘do’. Coordinating Puppet to take hosts out of the load balancer, deploy the new version of the application and restart everything can be done, but that is outside the sweet spot of the tool. Capistrano is better suited to these tasks.

Off Into the Sunset

Finally, sunsetting should be based on analysis of the services and the hardware required to provide them. Puppet’s configuration management correlates services with the hosts, removing exploration and guess work. Someone still has to make a decision, but with a fully managed system these decisions can be made faster with more certainty and less risk. Confidence is a good thing to have when cutting the power.

Use Your Judgement

So from my perspective, the decision is based on the timescale and to some degree a conceptual separation of the OS level systems and running applications. Use tools like Puppet to build up the system, get the applications started and general system management not disruptive to the running services. Use tools like Capistrano to orchestrate ongoing application deployment and change management. If your deployments are simple (PHP anyone?) you might get by with just Puppet or if your systems and services aren’t varied or complex you might be able to just roll them out with Capistrano. You can probably also cut a board in half with a claw hammer or hammer a nail with a hacksaw in a pinch.

Build IT, They Will Come

In conclusion, this is an exciting time to be building web infrastructure. The pressure to deliver applications on infrastructure, instead of CDs, is driving innovation. Scalability is best achieved through automation and the implications of virtualization towards that end are just starting to be realized. The goal is to get to a fully automated, fully flexible infrastructure where horizontally scaling a service is trivial, hardware failure has minimal impact and carrying the pager is a pleasure. All services are described as code, provisioning is automatic, deployment is on demand and sunsetting is a non issue. The tools to support automation are evolving, but taking fullest advantage is also dependent on process and culture. And that’s where I’ll start next time…