Agile Web Operations

Photo by furryscaly

In my last posts, I've introduced you to the IT Process Institute's "Visible Ops Handbook" which presents IT managers and system administrators with a clear road map of how to adopt best practices within their organizations and how to work smarter instead of harder. The first three phases outlined include change management, configuration management, and build processes. Today, I want to share with you their keystone practice that really ties all these pieces together in a single whole that will give your company the competitive edge in today's agile business world.

Photo by benchun

By now you should have a better understanding of how your data center is setup and exactly what's all in there. You've been carefully monitoring changes to this environment and are ready to take it to the next level. The "Visible Ops Handbook" introduces the concept of "production fuses" : when things go wrong with a fuse, you don't try to repair it; instead, you simply replace it with a working one right out of the package. Not only does this require less downtime, but you can be pretty sure that the new fuse will work as expected (and not accidentally burn down your entire home!). Your confidence in the correct functioning of the new fuse is directly related to how well you've been monitoring changes on that production fuse. If some engineer has made undocumented fixes or "improvements" to it (which is probably what caused it to short out to begin with), then the pre-fabricated fuse you replace it may cause unintentional side effects and further headaches. Replacing instead of fixing is what all high-performing organizations do, giving them high server to sysadmin ratios, less unplanned work, and the ability to maintain manageable system configurations.

The next step to get a better grip on your environment is figuring out exactly what kind of production configurations you have running out there. If you've ever caught yourself walking through the data center and wondering just what in the hell those servers in the back corner are for, this phase will be quite the eye-opener. Now that you have changes to this environment under better control you can start to invest some time in assessing what you actually have.

"Catch & Release"

Photo by NARA/EPA via pingnews

Last week, I introduced you to "The Visible Ops Handbook" and their 4 Agile Steps to ITIL Compliance. While there is no silver bullet for your particular problems, these steps should serve as a good starting point. Today, I'd like to go into a bit more detail regarding what the Visible Ops folks call "Phase One" of getting a grip on your operations environment. Remember, this first step can take many months to roll out so if you want to do this in an agile fashion, start small and gain forward momentum (and the rest of your team's confidence) with early, measurable successes.

Photo by shaggy359

Anybody in operations that wants to gain more control and understanding of their environment has heard of the IT Infrastructure Library (ITIL). This set of concepts and techniques introduced by the UK's Office of Government Commerce in 1980s heavily borrows from the ideas outlined in IBM's "Yellow Books" by Edward A. Van Schaik (and later merged into the single volume "Management System for the Information Business"). While such works can be sorted out and implemented by a skilled IT department, what about the smaller companies that want to start off on the right foot and not wait until they begin to grow by leaps and bounds to implement senseful IT practices? Enter "The Visible Ops Handbook" written by the IT Process Institute.

Photo by brettanicus

Given proper steering, teams must be empowered to decide and execute decisions on their own. This means that a team must have the proper make-up including designers and architects, application and database developers. Once a team commits to the story backlog, they must work together to ensure that all stories are delivered by the end of the sprint. If you've recently migrated from a department structured organization, making the whole team understand that they succeed or fail all together will take time.

Photo by Randy Son Of Robert

From the dawn of time, humans have always worked together as a team to overcome hardship and danger, and make the community stronger. Specialization naturally grouped people together to form hunting parties or food gatherers and later on governing councils and religious groups. This grouping together of dedicated, like-minded people forms the core of our success as a society today.

Fast forward to modern times. It's no coincidence that popular sports teams typically number between five and twelve people. Could you imagine a football team with 25 players per side? The communication and coordination of such a large group would be much to inefficient for such fast paced game play. To fans in the stadium, their reactions would seem slow and imprecise and probably not very entertaining. Yet this is precisely how many companies have organized themselves in the last century.

Photo by Mike Baird

The bigger a ship, the longer it takes to turn around. This old adage certainly applies to today's businesses, and if you're fighting to spin the steering wheel of a large vessel (by trying to change the work habits of hundreds of employees), you're in for a long battle. But don't let yourself get discouraged, because this is one instance where persistence literally pays off.

Start small but think big

You've worked for weeks on presenting a radical restructuring of your company's organization. Besides re-organizing team structures, you're also introducing ITIL policies to your operations department. You're convinced that these changes will allow the company to massively scale its customer base while increasing its operational prowess. Finally, the big day comes and your "Help me change the world" speech is applauded as divine inspiration. Yet weeks and then months slip by with no progress on any of your suggested initiatives. It seems your presentation was silently ignored and you're wondering how to get through to these people.

Have you ever found yourself in Sprint 4 or 5 without a single release under your belt? Is it because the new functionality involves a big database upgrade or depends upon coordinating three or four different departments? Not only does this kill motivation, but it's extremely risky to push out this mountain of code and configuration changes to production all in one release.

Photo by woodleywonderworks

Let's face it, compared to other engineering disciplines software development is just coming out of the stone age. Heck, I'm sure I'll get a lot of flak for even suggesting that software development is an engineering discipline (though I have to admit the way a lot of developers go about their work, calling it engineering does seem a bit of a stretch). Real life is seldom black and white, but I'd like to describe the two basic camps below.

Serial processing through departments for Release X.Y.Z

Most of us have worked at a place like this and, unfortunately, most of us still do. You know what I'm talking about - a traditional shop running with the waterfall process. Each department spends months working on a feature that ultimately, not many people are thrilled about releasing.

 development is finished when the calendar has advanced to a mystical date picked out of thin air months before

 after "finishing", the QA department can begin testing and will find the bugs (after all, this is their job, right?)

 operations is generally the last to know about the new feature resulting in emergency architecture meetings, more compromises, and even more delays