Agile Web Operations

Since Steve Souders's published "High Performance Websites" back in September 2007, a lot of people finally sat up and took notice of front-end web performance optimization techniques. Over a year later, you'd hope that most folks had figured out the basics like compressing content, using expires headers and reducing requests. Unfortunately, it seems that multi-core processors and bandwidth explosions have pretty much drowned out the awareness of this topic. Case in point, take a look at the search trends for mod_gzip vs. mod_deflate over the last 5 years.

        

Doesn't exactly represent a growing trend, does it?

Photo by furryscaly

In my last posts, I've introduced you to the IT Process Institute's "Visible Ops Handbook" which presents IT managers and system administrators with a clear road map of how to adopt best practices within their organizations and how to work smarter instead of harder. The first three phases outlined include change management, configuration management, and build processes. Today, I want to share with you their keystone practice that really ties all these pieces together in a single whole that will give your company the competitive edge in today's agile business world.

Photo by benchun

By now you should have a better understanding of how your data center is setup and exactly what's all in there. You've been carefully monitoring changes to this environment and are ready to take it to the next level. The "Visible Ops Handbook" introduces the concept of "production fuses" : when things go wrong with a fuse, you don't try to repair it; instead, you simply replace it with a working one right out of the package. Not only does this require less downtime, but you can be pretty sure that the new fuse will work as expected (and not accidentally burn down your entire home!). Your confidence in the correct functioning of the new fuse is directly related to how well you've been monitoring changes on that production fuse. If some engineer has made undocumented fixes or "improvements" to it (which is probably what caused it to short out to begin with), then the pre-fabricated fuse you replace it may cause unintentional side effects and further headaches. Replacing instead of fixing is what all high-performing organizations do, giving them high server to sysadmin ratios, less unplanned work, and the ability to maintain manageable system configurations.

The next step to get a better grip on your environment is figuring out exactly what kind of production configurations you have running out there. If you've ever caught yourself walking through the data center and wondering just what in the hell those servers in the back corner are for, this phase will be quite the eye-opener. Now that you have changes to this environment under better control you can start to invest some time in assessing what you actually have.

"Catch & Release"

Photo by NARA/EPA via pingnews

Last week, I introduced you to "The Visible Ops Handbook" and their 4 Agile Steps to ITIL Compliance. While there is no silver bullet for your particular problems, these steps should serve as a good starting point. Today, I'd like to go into a bit more detail regarding what the Visible Ops folks call "Phase One" of getting a grip on your operations environment. Remember, this first step can take many months to roll out so if you want to do this in an agile fashion, start small and gain forward momentum (and the rest of your team's confidence) with early, measurable successes.

Photo by shaggy359

Anybody in operations that wants to gain more control and understanding of their environment has heard of the IT Infrastructure Library (ITIL). This set of concepts and techniques introduced by the UK's Office of Government Commerce in 1980s heavily borrows from the ideas outlined in IBM's "Yellow Books" by Edward A. Van Schaik (and later merged into the single volume "Management System for the Information Business"). While such works can be sorted out and implemented by a skilled IT department, what about the smaller companies that want to start off on the right foot and not wait until they begin to grow by leaps and bounds to implement senseful IT practices? Enter "The Visible Ops Handbook" written by the IT Process Institute.

Photo by bertobox

They say you can't teach an old dog new tricks and this adage applies doubly to system administrators. After all, you pay these same people to protect your data and systems with tooth and claw, right? The problem is that sometimes they protect it too well. Ever needed quick access to data in order to make an informed business decision just to be told someone is on vacation or that particular system is down for maintenance until the end of the week? Are you still collaborating with remote teams via MS Excel? Have these added layers of security gotten in the way of your company's ability to grow and react to the customers' needs?

Photo by priittammets

After years spent working with Cfengine, Luke Kanies decided to form the company Reductive Labs in 2005 and Puppet, a long time idea and quickly stabilizing prototype, was born. He describes it as an open-source, next-generation server automation tool. Configuration files (called manifests) are written declaratively, and there is a client/server model for distribution handling. The configuration library, however, is what really distinguishes Puppet from its predecessor.

As promised in my last post about configuration management, I want to introduce you to one of the key Open Source configuration management players on the scene today - Cfengine. Embarked upon in 1993 by Mark Burgess, Cfengine has helped system administrators configuring and maintaining their servers in the data center for over a decade - bringing order to chaos and discipline to exhibitionism. This C based configuration management system can be manually executed, daemon driven and run by crontab and is controlled through a series of text-file configurations. Making use of a mostly declarative language, a single Cfengine statement can potentially cause hundreds of operations to be executed on multiple hosts across the network. Installation nowadays is as painless as yum install cfengine (Fedora Core/SuSE) or apt-get install cfengine2 (Debian).

Photo by vrysxy

The Velocity 2008 Conference hosted many excellent presentations and discussions concerning web performance and operations. Adam Jacob, of HJK Solutions, presented how his company goes about "Building An Automated Infrastructure". To briefly explain what an automated infrastructure is, let's think of servers and data as office buildings and automobiles. Would it make sense to begin construction of these without electricity, running water and roads? This infrastructure ties together our entire lives and greatly improves our standard of living. Correspondingly, basic IT tasks such as operating system installs, version control systems, configuration management and application deployment tie our servers and data together and greatly improve our ability to conduct business with customers.

I'd like to share my views on Configuration Management (CM) and the role it could play in your data center.