Agile Web Operations

Photo by priittammets

After years spent working with Cfengine, Luke Kanies decided to form the company Reductive Labs in 2005 and Puppet, a long time idea and quickly stabilizing prototype, was born. He describes it as an open-source, next-generation server automation tool. Configuration files (called manifests) are written declaratively, and there is a client/server model for distribution handling. The configuration library, however, is what really distinguishes Puppet from its predecessor.

Photo by chaosinjune

Evaluating hosting scenarios for our Ruby on Rails based web application, I finally had a good reason to dig deeper into "cloud computing". While I was already watching the evolution of Amazon's web services like S3 and EC2, a lot of offerings nowadays labelled as "cloud computing" were quite new to me. To better understand the range of offerings (and which ones might be suitable for hosting an application like ours), I tried to find a suitable categorization of cloud computing. What are the differences in all the offerings? What problems do they try to solve and how do they fit into my requirements for hosting our Ruby on Rails web application in a cloud?

As promised in my last post about configuration management, I want to introduce you to one of the key Open Source configuration management players on the scene today - Cfengine. Embarked upon in 1993 by Mark Burgess, Cfengine has helped system administrators configuring and maintaining their servers in the data center for over a decade - bringing order to chaos and discipline to exhibitionism. This C based configuration management system can be manually executed, daemon driven and run by crontab and is controlled through a series of text-file configurations. Making use of a mostly declarative language, a single Cfengine statement can potentially cause hundreds of operations to be executed on multiple hosts across the network. Installation nowadays is as painless as yum install cfengine (Fedora Core/SuSE) or apt-get install cfengine2 (Debian).

Photo by jpctalbot

The final building block of our introduction to agile is velocity. In addition to employing user stories to break down big features into manageable junks, maintaining a backlog for ruthless prioritizing, and story point estimates, velocity will help you find out what you can deliver in a week.

Photo by vrysxy

The Velocity 2008 Conference hosted many excellent presentations and discussions concerning web performance and operations. Adam Jacob, of HJK Solutions, presented how his company goes about "Building An Automated Infrastructure". To briefly explain what an automated infrastructure is, let's think of servers and data as office buildings and automobiles. Would it make sense to begin construction of these without electricity, running water and roads? This infrastructure ties together our entire lives and greatly improves our standard of living. Correspondingly, basic IT tasks such as operating system installs, version control systems, configuration management and application deployment tie our servers and data together and greatly improve our ability to conduct business with customers.

I'd like to share my views on Configuration Management (CM) and the role it could play in your data center.

Photo by HeavyWeightGeek

Estimation of Development Time is Hard

After discussing which issues we tried to solve by introducing agile practices to manage a remote development team, using User Stories to be able to compare requirements and building a Backlog for ruthless prioritizing I want to share our learnings about agile estimation of User Stories.

As you might have experienced, estimating the time required to build a certain piece of functionality is not easy. Either you have no clue and guess "3 days" or you have a feeling it will take around "4 hours". Now what if that 4 hour story takes you 12 hrs or more? Maybe you know you're too optimistic so you conservatively estimate 8 hrs. But it still ends up taking you 12 hrs. In fact, the longer you've been a developer the more you'll come to realize that your time estimate is hardly ever right!

Photo by Mike Baird

The bigger a ship, the longer it takes to turn around. This old adage certainly applies to today's businesses, and if you're fighting to spin the steering wheel of a large vessel (by trying to change the work habits of hundreds of employees), you're in for a long battle. But don't let yourself get discouraged, because this is one instance where persistence literally pays off.

Start small but think big

You've worked for weeks on presenting a radical restructuring of your company's organization. Besides re-organizing team structures, you're also introducing ITIL policies to your operations department. You're convinced that these changes will allow the company to massively scale its customer base while increasing its operational prowess. Finally, the big day comes and your "Help me change the world" speech is applauded as divine inspiration. Yet weeks and then months slip by with no progress on any of your suggested initiatives. It seems your presentation was silently ignored and you're wondering how to get through to these people.

Photo by mikefats

So far, I've talked about how I went for Introducing agile practices to manage a remote development team as well as User Stories - Making Sure Your Customers Get The First-class Seats. While User Stories are a good start, enforcing ruthless prioritization of these stories can really streamline your development processes.

Priorities get mixed up when you're doing to many things in parallel

One of the worst things about dealing with too many issues at once is that every one of them seems to be the most important one when you're looking at it by itself. The reason for such misjudgment is that while you discuss one issue via chat, phone or email, you often don't see the whole picture. Before you know it, you're constantly changing priorities and generating a lot of context switching in development.

Starting more work does not lead to faster results - on the contrary!

If you keep talking constantly about all your issues, you tend to think that everything is being worked on in parallel even if you have only one developer. So your expectations rise to unrealistic levels and you tend to push harder (often to the misfortune of both product quality and your team's morale).

Have you ever found yourself in Sprint 4 or 5 without a single release under your belt? Is it because the new functionality involves a big database upgrade or depends upon coordinating three or four different departments? Not only does this kill motivation, but it's extremely risky to push out this mountain of code and configuration changes to production all in one release.