What DevOps Is Not

This is a guest post by R.I. Pienaar (@ripienaar)

Many blog posts have been written about what DevOps is or what it means to specific people. While these blog posts are fantastic and well worth reading, there still seems to be a lot of misconception around its meaning.

I’d like to point out some of the misconceptions people have about DevOps.

DevOps is just PR for what we’ve always done

There is an element of truth in this. In many cases, there were teams who did what we call DevOps today.

What we are trying to do with the DevOps movement is create a focus point where new people can easily discover blog posts, mailing lists, like minded individuals and even books and conferences that help spread the word about how we think teams and individuals should work.

Contrast that with just searching the internet for guides or books on good Systems Administration practices. They do exist, but their scope tends to be very broad with a lot of background information and deep tech. This makes them hard to digest for people who aren’t that interested in being a full-time Operations guy. Now, blogs and twitter provide an easy and immediate way to find focused material you need for the task at hand.

The DevOps movement is trying to fill that gap of softer issues often overlooked in all the tech and tool talk.

In the past, turning a newly graduated engineer into a process savvy Operations expert with technical know-how took 10 to 15 years. As this pace is not sustainable, we need to adopt or die at this point. It’s my hope the DevOps movement will create a pool of information, communities and support groups that will one day help lower the entry bar.

DevOps mean giving developers the root password

Many people seem to think we are promoting a world where developers and system administrators all just run wild on the production systems and that it will somehow work out for the best. This is not what we are promoting at all. First, we are promoting closer cooperation between teams. Most companies will still have a development team and an operations team. DevOps are those individuals and processes that build the bridges between these teams.

Some developers might gain access to production in order to support their software but, unless you believe this will work for you, it doesn’t mean handing over the root passwords. Instead, it might mean a carefully constructed sudoers file or wrapper scripts that help your developers do what they need while still maintaining your needs for audit logs and knowledge of what is happening on the servers.

Discuss the needs of the developers and approach it with an open mind of how you could enable them to do what they need in a way that’s least disruptive to everyone involved. Tools like sudo and friends are well known and trusted. Embracing these tools means you can enable trusted individuals to perform trusted actions like restarting web servers during deploys without giving them the ability to restart just anything they want.

Share information with your counterparts in development. Give them network diagrams and spend lunch hours with them. This is how you’ll learn to know both the people and upcoming release work in a friendly, constructive way. Together, both teams are responsible for the end result. These simple actions slowly change the situation where System Administrators feel they catch all the flack for downtime and makes it so that all stake holders – both teams – become responsible for the overall stability, availability and performance.

All Systems Administrators should write code and all developers need to rack servers

People think now that we’re all able to do each others work we can just send a developer down to the datacenter. How will that ever work?

The reality is that not all Systems Administrators or Developers care to know in detail what’s happening on the other side of the fence – that’s ok. You will have teams of specialists in storage, networking, security and datacenter building. These teams might take the developers on tours of their work, what they do and why it matters but, ultimately, you will need those experts.

You will also have lots of developers who have limited interest in the systems side of the world and that’s ok too. You don’t need to know how to configure a Cisco router to build an AJAX user interface.

What you will have are a few individuals who move freely between teams – they might have stronger background in one than the other – but they will work together and build bridges, carrying information and knowledge between those two areas of work.

We don’t need Systems Administrators – we’re DevOps and we use the Cloud!

You might not need a load of guys building storage networks and redundant networking infrastructure, but you should still get an expert consultant for the initial setup or have in-house systems expertise.

Over decades, professional Systems Administrators have learned the value of backups, strong security policies, documentation, monitoring and countless other aspects of system architecture and networks. There are a lot of tasks that get done almost as second nature now that contribute to a better whole. A Systems Administrator will know what to look for in vendors – even ones to which you’re considering outsourcing. The *aaS landscape is incredibly busy and full of people with very glossy wording and PR campaigns. You need someone to evaluate and help pick the right partners.

Developers who have always had a different set of experiences, values etc. are often just not aware of these things and don’t know to keep them in mind. The value in cooperation is that developers can become aware and equally Systems Administrators can become aware of issues they didn’t consider before.

There have been several cases over the last few years of startups shutting down or experiencing extended downtime due to:

• Backups that didn’t get done
• Monitoring that didn’t cover the machine performance or had too much focus on application metrics (masking nasty scaling issues)
• Too trusting networks allowing a simple config change in development which destroys production data.
• Scaling strategies that are ill conceived or built with unrealistic expectations from networks and servers

In many of these cases, the Systems Administrator, who you’re trying to put out of a job, would have spotted these problems early on. He might have been a bit of a bastard explaining it to you, but a well run system that has the correct foundation right from the start functions a whole lot better in the long run. The cloud removes a lot of work that Systems guys have done in the past, but it doesn’t remove the need for solid architecture that covers all the bases that good Systems Administrators know as second nature.

The cloud brings with it a lot of new issues: performance monitoring and debugging of changing system behavior is very difficult and you need experts who understand how systems interact to debug performance issues. Planning scaling strategies, backup strategies and inter cloud HA are very complex systems problems. You will absolutely need architects who know how systems and networks work at a deep level for that.

What DevOps and automation brings to this is allowing your systems guys to focus on improving key core systems. Since they’re not continually interrupted and bogged down with easily automated, boring tasks, they have sufficient time to correctly build important components.

DevOps places the responsibility of managing an aspect of your system with the right people. If your code is changing daily, the Operations team isn’t the right group to deploy and support it. They become bottle necks and cause friction.

Stronger cooperation between teams means your developers will be enabled to do their work while Systems Administrators do theirs. It might mean developers going on call for some aspect of the system, but in an ever changing, agile landscape the ones developing and doing the daily deploys are definitely the best ones to support it. Embrace the change and enable everyone.

DevOps require you to use tool xxxx

DevOps is a cultural movement and covers a lot of soft issues. Even as the technology landscape drastically changes, our tool chain has been invalidated by the need for ever more dynamic and agile infrastructures, short lived computing instances, virtualization and other related technologies.

The modern network needs to be agile. We need to be able to provision servers in minutes rather than weeks or hours, and, sometimes, these machines will only remain active for just a few hours. We need to have monitoring, inventory and other related services that can cope with this level of dynamism. Internet scale user bases has also made it common place for even reasonably small shops to have over 500 nodes, so the man power required is immense.

These factors will force automation to the forefront, and that’s why there’s so much focus on various automated server building tools, configuration management tools and command and control tools. However, it’s not about the tools – it’s about the problem. You need an automated infrastructure, and the choice of tool always comes down to what works best for you, your organization and its needs.

We are a large enterprise and the vendor tools cannot be automated, we cant practice DevOps

DevOps isn’t an all in or all out thing. You can liberally apply the 80/20 rule depending on your needs. Many people say if you have to do anything by hand you’re doing it wrong. But, in many cases, the effort to automate the last 20% is simply not worth it. Or, as the example says, the applications just aren’t designed to be installed in an automated fashion.

Many people who manage servers using configuration management tools will manage 100s or possibly 1000s of resources per machine. Even if you just start small and manage your sudoers file you’re already doing one thing better than before. Automation is a lifecycle not an event; starting small and improving over time is perfectly acceptable.

Starting with one thing, learning and exploring ways to apply what you’ve learned to other aspects of your architecture has value and is encouraged. You can have a ‘base’ machine that has just your users, backup agents, monitoring agents, etc. deployed. Then, on top of that, hand install your enterprise tool. Even if you don’t automate the last bit, you’ll still reap huge benefits in managing the base parts of your server state.

We need to employ ‘DevOps’

DevOps is not a job title or a specific role. Your organization probably already has senior Systems guys and senior Developers who have a lot of the traits needed to work in the way that DevOps promotes. With a bit of effort and help from outside consultants, mailing lists or conferences, you might easily be able to restructure your business around the principals we propose without employing new people – or losing old ones.

There is no such thing as a DevOp; it is not a job title. Feel free to advertise for people who work with a DevOps mentality, but there are no DevOps job titles.

Often a good person to consider in the role as a bridge between teams are generalists, architects and Senior Systems Administrators and Developers. Many companies in the past decade have employed a number of specialists – a DNS Administrator is not unheard of. You can still have these roles, but you’ll need some generalists who have a good background in multiple technologies. They should be able to champion the values of Simple systems over Complex ones, and begin establishing automation and cooperation between teams.

Conclusion

There are many more misconceptions that should be addressed in time. The ones above are what I hear most often. The message should be that we are not trying to put people out of work. We are not trying to force people to become who they’re not and we are not advocating only flat organizational structures.

We are promoting cooperation, tolerance and shared responsibilities between Development and Operations teams. We believe that by sharing the pains and rewards of running a successful business across these teams, your business will become even more successful and your staff happier and more effective.

About the author

With more than a decade experience in very large, multi-national companies, R.I.Pienaar is a Systems Administrator currently running his own consulting company in London.
He’s the author of several open source projects, including e.g. “The Marionette Collective” server orchestration and a host of others.