Agile Web Operations

Agile @ NetDoktor

Dan Ackerson — Sat, 20 Mar 2010 07:51:54 +0000

I’ve been in the process of introducing agile over at NetDoktor for over a year. I really like the sound of “in the process of introducing agile”. It’s kinda like the permanent Gmail Beta (or Flickr Alpha). It means there will never really be a “final” agile process here and that’s a great thing! Why? Because the whole point of agile is adapting. “Adapt or die” is about the most basic business tenet I can imagine.

Click here to read the full post!

Stop. Reflect. Adapt. The 3 Steps to Stop Writing Bad Code

Matthias Marschall — Wed, 17 Mar 2010 06:07:08 +0000

Writing software that doesn’t suck is hard – even for the pros. The problem doesn’t lie in solving a hard problem, but in creating a solution which is easy to understand, robust, and easy to change.
A lot of problems in teams and organizations stem from bad code. Bad code ruins the motivation of your team, slows you down to a crawl and drags you down, deeper and deeper into the fire fighting vicious cycle.

Why do we write bad code?

Let’s try to understand a few reasons why we continue to write bad code, even though nobody wants to do it.

In my experience, a lot of developers always feel under pressure to deliver faster. Of course, there is constant pressure to finish a product faster. And of course, everybody should work towards the goal of getting features out to the customers asap. The issue is that too many software developers cut corners when trying to speed up. They don’t write automated tests, they don’t refactor their code after finishing an initial draft of working software, and they claim they’re done even though only the most basic flows are working (and they never took the time to thoroughly test border cases).
We all know where this leads to: Dropping half assed features into QA creates a myriad of bugs and a lot of hassle fixing the left overs of the rush. Not having any automated tests makes you play russian roulette every time you touch your code, and adding and patching a draft version of the code makes it worse with every bug fixed.
Everyone feels it: Your code is going down the drain.

How can we break that vicious cycle?

If you’re stuck with a shit load of crappy code you have to stop the downward spiral. The recipe is simple, but, while it will get you into the right direction, and it will lead you a path of evolutinary code and culture improvements, it’s not a quick fix nor a silver bullet. However, it’s your only chance. There are just three steps:

Stop. Reflect. Adjust.

Stop

First of all, you’ve got to realize you can’t go on like this. You’ve got to stop the line. This sounds very counter intuitive given the fact that you’re late already. But, only by stopping can start improving.

Reflect

Get everyone into one room. Ask them for noteworthy events within the last week. What did they experience? What got in their way? Make it clear this is not a blaming game, just a fact-finding mission. If you’ve collected enough, give everyone five points to distribute among the issues: Let them put as many of their five points on any issue they think should be addressed first. With this exercise, you get a shared understanding of the trouble spots and a first assessment of what’s worst. This is your input for the next step.

Adapt

Take the one issue which got the majority of the rating points in the reflection meeting. Make its resolution top priority for the coming week. This will signal to your team that you take their problems seriously, and you won’t continue repeating the same mistake. Address their concerns and show them how to get better – one step at a time.

Repeat

Repeat the reflect and adapt steps every week. Make sure, you address the worst issue every week. Interestingly enough, in the first weeks most issues won’t be considered to be in the area of the team’s influence. As the team lead, you have to make sure to get them resolved. Over time, however, the team will become more self confident and tackle more and more issues on their own. Now you’ve created an upward spiral where you get better and faster every week.

Have you ever experienced similar situations? How did you deal with them? Let us know in the comments below!

20 DevOps guys you should follow

Matthias Marschall — Thu, 04 Mar 2010 09:42:01 +0000

DevOps is an approach to bridge the gap between agile software development and operations. The DevOps tribe is a growing group of people practicing a new way of combining development and system administration for more speed, quality, revenues, and fun.

The DevOps Tribe

Here is a list of some of the most active guys in the DevOps realm. It’s by far not complete and in no special order. But if you want to learn what’s going on with DevOps, you should follow those folks.

Patrick Debois
Inventor of DevOpsDays – the conference that brings development and operations together. While many of us were thinking and talking about DevOps, we didn’t even have a forum before Patrick launched DevOpsDays 2009 in Ghent, Belgium.
Blog: Just Enough Documented Information
Twitter: @patrickdebois
Patrick’s DevOps Intro Video

Damon Edwards
President of DTO Solutions (a consulting group focused on automated infrastructure and IT process improvement)
Blog: dev2ops delivering change
Featured DevOps post: What is DevOps?

John Allspaw
At the moment, he’s VP of Technical Operations at Etsy. He’s worked at Flickr, Friendster, InfoWorld, Salon, Genentech, Volpe National Transportation Center, and a bunch of other places as a consultant from time to time.
Blog: Kitchen Soap
Twitter: @allspaw
Featured DevOps presentation: 10 deploys per day – DevOps cooperation at Flickr

Lindsay Holmwood
Lindsay Holmwood is a sysadmin/developer from Sydney, Australia.
He’s the creator of cucumber-nagios, Flapjack, Visage, and Gastro.
Blog: auxesis musings
Twitter: @auxesis

Andrew Shafer
Engineering Lead at Cloud Scaling. He is interested in public speaking, cloud computing, systems management, startups, Agile, lean, Ruby, Clojure, dynamic programming languages, motivation or whatever random thing seems interesting to you.
Blog: stochasticresonance
Twitter: @littleidea
Featured DevOps Post: Puppet, Chef, Dependencies And Worldviews

Paul Nasrat
Paul is a software developer and systems engineer from London, UK.
Blog: Pauls Wibblings
Twitter: @nasrat
Featured DevOps Post: Puppet Camp 2009

Mick Pollard
Mick’s current interests in linux/sysadmin are system deployment (pxe/kickstart) and configuration management ( puppet ). He’s also interested in all things hosting and scaling of webapps.
Blog: lunix
Twitter: @_lunix_

James Turnbull
James is an Australian free software and open source author, security specialist, and software developer contributing to the qpsmtpd SMTP daemon, the Puppet configuration management tool, and the Facter system inventory tool. He wrote books like “Pulling Strings with Puppet” and “Pro Nagios 2.0″. James lives in Melbourne, Australia.
Blog: Kartar.Net
Twitter: @kartar
Featured DevOps Post: What DevOps means to me…

Koen Van Exem
A DevOps from Mechelen, Belgium.
Blog: InxIn
Twitter: @KoenVanExem

Gildas Le Nadan
Gildas specialises in IT Operations Coaching/Infrastructures, Open Source, Business Continuity/Disaster Recovery. He is from Lille, France.
Blog: On IT Operations And Infrastructure
Twitter: @endemics
Featured DevOps Post: Self Documented Agile Infrastructure

Chris Read
Chris currently works for ThoughtWorks as a Principal Technical Consultant and Infrastructure Specialist. His specialties are Unix (any flavour) and networking. He’s done his time as a developer, but finds the path of the Sys Admin much more rewarding…
Blog: Chris Read
Twitter: @cread

Kris Buytaert
Kris Buytaert is a long time Linux and Open Source Consultant doing Linux and Open Source projects in Belgium , Europe and the rest of the universe.
Blog: Everything is a Freaking DNS Problem
Twitter: @KrisBuytaert

Stephen Nelson-Smith
a Technical Manager and Devop based in Hampshire, UK and author of Agile Sysadmin
Blog: Agile Sysadmin
Twitter: @lordcope
Featured DevOps Post: What Is This DevOps Thing, Anyway?

R.I. Pienaar
Systems Administrator, Consultant, Linux Guy, Automator, Ruby Coder. Currently in London, UK.
Blog: www.devco.net
Twitter: @ripienaar

Julian Simpson
Julian is helping to deliver working software, one continuous integration build at a time.
Blog: The Build Doctor
Twitter: @builddoctor

John Willis
Vice President of Training & Services at opscode, the company behind Chef.
Blog: IT MANAGEMENT AND CLOUD BLOG
Twitter: @botchagalupe
Featured DevOps Post: Does Automation Replace Humans?

Marcel Wegermann
Sysadmin, web developer and test infected – Marcel currently works for it-agile in Hamburg, Germany.
Blog: Agile System Administration
Twitter: @marcelwegermann

John Arundel
Freelance sysadmin & systems integrator, devop, hacker, geek. Likes Puppet, Drupal, agile, and a lot of other cool stuff.
Blog: Bitfield Consulting
Twitter: @bitfield

This list is brought to you by:
Matthias Marschall
CTO of autoplenum.de – hands-on involved in development and operations of the site and always seeking to make the whole company more agile end-to-end.
Blog: Agile Web Operations (you’re currently reading it )
Twitter: @mmarschall

and

Dan Ackerson
Having worked as both a developer and an operations manager, he’s gotten all too familiar with the ever widening gap between what developers and customers consider “done”. That’s why he wants to help to bridge the deployment gap.
Blog: Agile Web Operations
Twitter: @danackerson

If you want to be listed here, just let us know through our contact form or write us a comment below.

Agile Links From The Archives

Matthias Marschall — Tue, 02 Mar 2010 21:42:33 +0000

One finding from our survey was that a lot of you want to read more about agile basics. As most of you haven’t followed Agile Web Operations since Day One, here’s a list of the top three posts about agile and kanban:

And last but not least, you might want to follow my five part series:
Introducing Agile Practices to Manage a Remote Development Team. Discussing User Stories, the Backlog, Story Points, and Velocity provide a good overview of basic agile management practices. Enjoy!

RSpec Tips & Tricks

Matthias Marschall — Thu, 18 Feb 2010 11:06:41 +0000

Throughout the last couple of days, I did a bigger refactoring of our Ruby on Rails application. As I changed quite a few moving parts, I covered everything I did with RSpec. It’s really an incredible feeling to have all your bases covered with automated tests when you finally start the manual regression test. Along the way, I came across a few things I wanted to share with you.

Structuring Your RSpec

When you’re writing specs you want to bring order to them sooner or later. I really use nested describe-blocks a lot. Let’s look into a Controller spec to see what I do:

Every controller spec starts with a describe block like this:

describe QuestionController do
 
end

For spec’ing out individual actions in the controller, I usually create one describe block for every public method in that controller:

describe QuestionController do
	describe "index" do
  end
end

Inside that describe block I put all the it-specs themselves. Usually, I start by writing only the expectation and later I fill it out:

describe QuestionController do
	describe "index" do
		it "should redirect to 404, if no questions are found"
		it "should prepare a list of ten questions"
		it "should redirect a not logged in user to the login page"
  end
end

Running specs from the command line

I use RubyMine for my ruby development but despite it’s capability of running specs within the IDE I prefer to run them separately on the command line. For running only the part you’re currently working on the -l parameter comes in handy. You can pass it any line number of an it or describe block and it runs only what’s there. This is especially cool for describe blocks as it runs all specs inside that block – and only those.

If you structure your specs like me using nested describe blocks, this is really nice.

Agile Web Operations: What do YOU want it to be?

Matthias Marschall — Tue, 09 Feb 2010 21:00:45 +0000

For nearly two years Dan and I have shared our experiences and ideas about agile development and system administration. With every post we hoped to be helpful, and maybe some of them even were…

Now, as we approach 500 subscribers, we would like to ask you, our dear readers, how we could help you to become even more agile and have more fun doing your job.

Please take our short survey at: http://www.surveymonkey.com/s/LJVMRG9 and tell us about your situation, your goals and how you would profit most from our experiences.

We’d love to hear from you!

Dan & Matthias

Pragmatic Personas: Concrete Examples of Your Users

Matthias Marschall — Tue, 02 Feb 2010 09:46:06 +0000

Jeff Patton’s talk at agile 2009 about Pragmatic Personas is quite interesting. I’ve seen talks about personas way back at agile 2007 already, but, at that time, I found them quite “bulky” to use. In pragmatic personas I see more value.

What is a Pragmatic Persona?

Jeff defines a pragmatic persona by having a name like “Jeff, UX Expert”, describing the current situation of the persona, an about section describing goals and needs of the persona and a section describing how our application could add value for the persona.

Baldin, the homeless dude

Situation:

43 years old
has no home
ran out of money a long time ago

Goals:

Wants to find the best place to stay for the night

Value:

Our webapp could show him the occupation status of the various bridges in the area.
He could reserve a place

Personas Act as Concrete Examples

When you’re doing TDD, you formulate concrete examples to test your code. You test your divide method with examples like divide(0, 0), divide(0, 1), divide(4, 2), etc.
Personas can act as concrete examples for your user stories and your automated acceptance tests. They will never give you the full picture of all your users, but if you define them wisely, you can come up with the critical variations (like finding the border cases for your test cases e.g. divide by zero).

A Persona Tells You What You Know About Your Users And What You Don’t Know

One interesting aspect of using pragmatic personas is that by trying to fill out situation, goals and values you find out what you don’t know about your users. If you have trouble describing a sample situation or the goals of a specific user, you should go and find out more about it. This way, pragmatic personas reduce the risk in your product development by pointing out your weak spots before it is too late.

Only using one “persona”: the user (as most of us usually do) is much worse than coming up with some real world examples of who is using your product and why. Give it a try, you won’t regret it!

Tailoring Your Munin Installation

Matthias Marschall — Mon, 21 Dec 2009 20:31:07 +0000

After following Dan’s tutorial on installing munin on your servers, you already get the benefits of munin’s default plugins. You have graphs showing your CPU, RAM, I/O, as well as MySQL, Exim, and quite some other stats. But most of the time you run some additional software which you also want to montior.

Install Some Custom Munin Plugins

In our case, we have a full Ruby on Rails stack featuring nginx front end web server, HA Proxy load balancer, a cluster of thin instances for the Rails application, memcached for caching page fragments and a MySQL database backend.
No matter how your stack looks like, you’ll find some good munin plugins at MuninExchange. Some of them are pretty basic (or not working as required) but give you a great head start. Just grab one of them and put it into the /usr/share/munin/plugins dir on any munin node. Make it executable (chmod 755 thin_memory) and enable the plugin by symlinking it to /etc/munin/plugins (you must restart your munin-node after adding new plugins).

Alternatively to manually symlinking your plugin, just call root@node:/usr/share/munin/plugins# munin-node-configure --shell to do it for you. It will only link plugins compatible with your configuration (see the munin-node config docs for details)

Test Your Custom Munin Plugins

You’ve got basically three ways of making sure your plugin works:

Run the munin plugin script directly:
This is the most basic way to ensure, that your munin plugin returns reasonable values. Just execute the script directly on your munin node.
```
root@node:/etc/munin/plugins# ./thin_memory
```
Your script should return something like this (depending on how many thin instances you run on which ports):
```
thin_8000.value 180
thin_8001.value 178
thin_8002.value 175
thin_8003.value 169
```

Use munin-run to execute the plugin locally:
The next stage is to test whether the munin plugin still returns reasonable values when executed as a munin-node process:

root@node:/etc/munin/plugins# munin-run thin_memory --debug

This should return something like this:

# Saving mysql*->env->mysqlopts = --defaults-extra-file=/etc/mysql/debian.cnf...
...
# file: 'thin_memory'
# Checking thin_*...
# Checking thin_*...
# Checking thin_*...
# Checking thin_*...
# Want to run as euid/egid 0/106
# Running as uid/gid/euid/egid 0/106 106/0/106 106
# DEBUG: About to exec "/etc/munin/plugins/thin_memory"
thin_8000.value 180
thin_8001.value 178
thin_8002.value 175
thin_8003.value 169

Run munin-update at your munin server: The final stage is to manually invoke the munin server process to see, whether it is able to collect data from your munin plugin remotely:

root@munin-server:~# /usr/share/munin/munin-update --debug --nofork --stdout --host node.example.com --service thin_memory

Watch out for lines like these:

...
Dec 17 18:41:31 - [17775] Fetching service: node.example.com->thin_memory
Dec 17 18:41:31 - [17775] Updating /var/lib/munin/reference/node.example.com-thin_memory-thin_8000-g.rrd with 180
Dec 17 18:41:31 - [17775] Updating /var/lib/munin/reference/node.example.com-thin_memory-thin_8001-g.rrd with 179
Dec 17 18:41:31 - [17775] Updating /var/lib/munin/reference/node.example.com-thin_memory-thin_8002-g.rrd with 175
Dec 17 18:41:31 - [17775] Updating /var/lib/munin/reference/node.example.com-thin_memory-thin_8003-g.rrd with 169
Dec 17 18:41:31 - [17775] Fetched service: node.example.com -> thin_memory (0.08 sec)
...

And, last, but not least, there’s good, old telnet which you can use to contact and query any munin-node:

me@anywhere:~$ telnet node 4949

You may use list to see, which services are monitored at your node, and fetch thin_memory to get the current values. Just type help to see what else you can do there.

Customizing your munin installation with additional munin plugins is quite easy and straight forward. You just need some scripts from MuninExchange and enable them for use via munin-node. The various ways of debugging your plugins really help to get your monitoring up fast.

Sub-optimization Kills Customer Value

Matthias Marschall — Tue, 15 Dec 2009 20:22:02 +0000

When we start optimizing our processes, it happens quite often that we only optimize our area of influence instead of addressing the whole process of creating customer value. When we’re responsible for a software development or an operations team, we tend to optimize the process of our team. We adapt agile practices and our teams performance seems to skyrocket. But even if we’re that successful, it might do more harm than good. We might flood the QA team with features, which they have to sign off, creating overload on their end. Or our operations environment might be the greatest part of our organization, but there are no new features or necessary bug fixes arriving for deployment.

We Need to Think End-to-end

In both scenarios the customer does not benefit from our optimization. We sub-optimized our area of responsibility without taking the complete company with us. Only if everyone involved in creating value is working together and bottlenecks are removed from all stages do customers truly benefit.

Finding and Removing Bottlenecks

For being able to optimize the whole, we’ve to look outside of our home turf. We have to find out how customer demand reaches our organization, how it flows through the various departments and how the demanded value is finally delivered to our customers. One tool to use for such analysis is Value Stream Mapping.

When we’ve identified the biggest bottleneck in our end-to-end flow, it’s time to start removing it. If it lies in our area of responsibility, that’s rather easy. But if it is outside of our direct reach, we need to get into office politics – we need to convince others that they have a problem and that they have to fix it. That’s usually not too easy. But, again, only if we optimize the whole instead of sub-optimizing parts of the value creation flow inside our organization will the customer truly benefit.

Agile Is About Feedback, Not About Fancy Practices

Matthias Marschall — Wed, 09 Dec 2009 16:13:34 +0000

Too often people complain that to become agile they need to start using iterations, fancy story points and time boxes even though it simply does not fit the way they work.
But, that’s not true. Agile is much simpler than that. And much harder. In essence, agile is about fast feedback. But the feedback needs to be relevant.

Real Customer Value

The most important thing to make feedback relevant is to get it from the right people. Especially in design questions, everyone seems to be an expert. But, in the end, only the end users count. Are they able to get value out of using your product? That’s the only thing which really counts and that’s usually the feedback which is hardest to get.

The Art Of Knowing…

One way of getting that feedback is demoing working software at the end of an iteration. By doing so, you ensure that you get feedback at least every couple of weeks by at least customer representatives (if no real users can take part in your demos). But there’s an even faster way of getting direct feedback from your real users. By using click tracking, session recording and A/B testing (Google Website Optimizer, A/Bingo), you get relevant feedback constantly. How can you possibly get more agile? And this has nothing to do with fancy, magic agile practices…