GoTRISI

No clear picture of your dream candidate

Usually, people slap together a list of things they need a candidate to know or to have experience in. But this isn’t enough. Too many companies are already searching similar profiles on that level of detail. Imagine your dream candidate: What is his current situation? What is he doing right now? What are his fears and dreams? Only if you zoom in and create a very vivid and detailed picture of your dream candidate, will you be able to address the right people.

No compelling message

A job ad is like a sales letter. It needs to address the problem you solve for your candidate (find a great job) and it needs to show why your job is the best solution. Additionally, it needs to address objections the candidates might have reading your job ad (“will they keep my inquiry confidential?”, “will they offer fair payment?”, etc). Furthermore, it helps to add some testimonials to your job ad: Let your team mates describe how switching jobs was for them. And, last but not least, you need to create a compelling reason why your job is unique and not just like any other engineering job out there.

Searching the wrong places

The best job ad is worthless if it doesn’t reach the desired candidates. Instead of just posting it to general purpose job sites like monster.com, you have to chase down your dream candidates and find the places they hang out. Failing to do so means getting flooded by the wrong candidates and wasting a lot of time and money. If you’re looking for a ruby expert, who regularly shares his thoughts and code, you might find them on twitter, github, or stackoverflow.

Poor Handling of Applications

Too much can go wrong in the final stages of the hiring process. The best way to lose great candidates is waiting too long. Waiting too long to invite them for an interview, waiting too long to give them feedback, waiting too long to make an offer. When you start the recruiting process you need to define the exact procedures for following up with candidates. Who needs to decide what? How fast can they make those decisions, etc. If you communicate a clear process with well defined timing, chances are your candidate will be engaged and stay tuned.

Failure to close the deal

After going through all the necessary interviews, you’ve lined up a dream candidate for hiring. This is a critical step in the process. You need to make a compelling offer and dissolve all objections the candidate might have about signing up with you. And it’s time again to re-emphasize your uniqueness to stop the candidate shopping around for comparable offers. Only if the candidate knows everything about your company which helps them make a positive decision, will you be able to close the deal.

What are your worst experiences when trying to hire someone or when trying to join a company? Please share your horror stories in the comments.

No related posts.

Ken Teegardin

While it may be hard to believe, this is how the majority of software development has been done for almost half-a-century. The older the company, the better the chance they follow this exact release cycle. There are annual budget meetings to determine which departments and projects will gain the largest developer pools. It’s excruciating to watch these behemoths try to introduce agile into such an environment and I believe budgets are toxic to agile development!

Yes, we know that budgets have an extremely important place in every day business. This is how we plan our operating costs, growth and future development projects. However, when budgeting meets project management, there is no way to be agile. Take the following statement: “If we aren’t allocated these 10 resources tomorrow, there is no way we can finish this project within 12 months.”

Try to wrap your head around that sentence. Finishing the project says nothing about a customer release. And what exactly are these “10 resources”? Mainframes executing COBOL statements? Project managers frantically waving Excel spreadsheets? Developers attending specification meetings? Can we so easily secure next year’s release just by offering up these resources like lambs upon the project manager’s sacrificial altar?

My almost instantaneous answer to such a hollow statement is “Given these 10 resources and 12 months of development, you will never be able to ship a quality product.” And the reality is probably even more bleak – they won’t be able to ship anything at all!

Think about 12 months from now. It’s so far away. We can take our time building a sound architecture, something that will be infinitely scaleable, meticulously documented and a joy to use. See? Scope creep has happened already without a single line of code being written!

If you aren’t challenged to regularly engage the user, to ship your code at least once a month, you fall into a false sense of security – a dangerous reality distortion field is emitted from your project that can even have detrimental effects to projects that rely on yours.

Not all desktop applications follow such a cycle. Google has pioneered regular release cycles on the desktop with it’s Chrome browser (4-6 week release cycles). Now, if only we could get a glimpse inside of their PM process.

What do you think of annual budgets and their effect on scrum?

Other posts:

• Why Excel Spreadsheets Hurt Project Management
• State of Development: Annual Address on How We Ship Software

benjamin-nagel

Outsource a software development project requiring 10 developers, an on-site team of 3 managers and 4 developers, involving a total of 4 external companies. Surprised that the shipped product is more complicated than you originally planned? You shouldn’t be.

Complexity is in the details

We’ve all encountered this phenomenon in software projects. If it seems too easy, it’s because you’re missing something. This is why we do Planning Games – to spend an extra hour carefully thinking about these “easy” stories – picking apart the plain vanilla requirements and finding some very important open questions.

As software engineers, we tend to overdesign and overarchitect. Now, offshore the project and get ready for the fireworks. External engineers have no clue about your business, and could care even less about the project’s success. On top of this, the channels of communication are constricted. Sufficiently insulated from reality, the offshore team is free to dream of the “perfect solution”.

Perfection meets reality

In a perfect world, nothing fails. And this is exactly how those consultants engineered the architecture. No fail-over servers, no backups, and no documentation.

Pro-tip: make the consultant reboot the server at lunch time every day of their last week. Make detailed bug reports of all the failures that occur and get them fixed before the consultant takes another job!

Stability through Simplicity

You know your business and, hopefully, what your customers need. I’ve done enough “in-shoring” of projects and architectures by now, to recognize the patterns. Over-implemented security measures, unnecessary servers and lots of external ISP support.

Cutting through the baked-in complexity of a project years in the making isn’t easy. But, if you focus on the end customer value, you can quickly get through to the real meat.

Other posts:

• Aegis: Role-based Permissions for your Ruby on Rails application

George Vnoucek

You know it within an hour of working with them. A special kind of sysadmin or developer that not only knows how to do their job, but really cares about doing it right. This is the person that makes you refactor that duplicate method or add that one last test. The kind of engineer that dots every i and crosses every t. David Lutz (@dlutzy) tweeted this last week “Of the IT professionals I’ve worked with, talented sysadmins and programmers, only a select few are worthy of the title Engineer.”, which really struck a chord with me.

I asked him what were the three common traits he thought these successful engineers exhibited, to which he replied “1. deep understanding of how things work. 2. thorough and methodical approach. 3. ongoing analysis of results”. I think he nailed this and I’d like to dig a bit deeper into each of these traits.

Deep Understanding

It’s one thing to be able to code to an API or install an OS from a CD. But, to really understand, on a system level, everything that’s happening in the background to make all the “magic” possible is a different level altogether. From taking measures to avoid race conditions in software, to ensuring an SCM based OS configuration for automatically bootstrapping new server instances, successful engineers employ these traits irregardless of time constraints.

For me, a deep understanding means decades of researching, experimenting with and testing of new technologies. I spent the first half of 2010 learning puppet and creating server configurations for not only my productive and staging environments, but the continuous integration server, wiki and blog. One year later, I’ve learned just as much about puppet (and my servers) as I did in those first intense six months. Do I have a deep understanding of puppet? Hell no.

Thorough and Methodical

So what about those pesky time constraints I mentioned. Unfortunately, they’re what make average developers bad developers. They are encouraged by management to “just finish it” at the cost of code quality, countless bugs and many late nights. How often have you heard a developer say “My manager says we don’t have time for tests.” ?

Successful engineers know that doing things right means taking the extra time needed to make sure that a feature or a service works exactly as expected. That means writing exhaustive tests and introducing monitors which ensure these things are up and running not only on the live site, but after every change made to the system.

Ongoing analysis

Just because it’s out in the wild, doesn’t mean an engineer’s job is done. In the constant pursuit of perfection, they analyze how it’s performing – does it react to load and other issues as expected? Poring over megabytes of log files and checking those Cacti graphs every morning is just an ingrained habit by now.

As a system matures, usage changes, requirements change – refactoring and upgrades need to be regularly applied to keep the system healthy and growing. Often times, it’s those little tweaks months after the release that make or break the future of a platform. A simple query cache tweak here, or a refined Apache configuration there and suddenly the application is fun to use again.

Successful engineers know the intrinsic value of these often behind-the-scenes and unrewarded “patches”. They keep on doing it because they take pride in their work, and desire to honor the profession of engineering and engineers everywhere.

Sharing ideas

Twitter is such an incredibly cool platform. Being able to casually share such deep insights with like-minded people instantly is truly amazing. It’s exactly this kind of brain candy that has me totally hooked.

Who are some inspiring, successful engineers that you follow on Twitter? Please share with @mmarschall, @danackerson or in the comments below!

No related posts.

The cool thing about doing geeky things like this is the ability to share the howto with other folks who can actually get just as excited. Amazon’s announcement of a 12 month free usage tier, plus the fact that this blog doesn’t run on sugar plums and candy canes, put us in the short list for migration. As Matthias has done the bulk of the joyent server setups & linode migrations, it was high-time I got my hands dirty. Without further ado, here’s how I migrated AWO to our first Amazon EC2 instance. Let’s get our geek on!

Choosing BitNami

Initial plans were to grab some debian-based Amazon Machine Image (AMI) and follow the various “howto config wordpress on lamp” posts out there, but Matthias asked if I was going to take a look at BitNami. I’d never heard of it before and, truth be told, it actually sounded like some kind of BitTorrent client to me. Digging in a little deeper, I soon understood what all the hoopla was about and I was sold on a nicely contained, pre-configured BitNami WordPress AMI. There’s even an AMI which automatically bootstraps an Elastic Block Storage (EBS) – how’s that for easy?

Just in case you don’t know what EBS is good for – a running EC2 instance doesn’t come with a disk for storing your data by default. Meaning if you plan on making changes to your AMI at runtime that you’d like to see after a reboot, you need to attach a disk (EBS in Amazon AWS speak).

Setting up a new EC2 instance on AWS

One of the catches for a 12 month, free EC2 instance is a new Amazon account, so be sure to register a new email address or you might get a surprise bill next month! And, yes, you will also need a valid credit card to open your account, because if your instance exceeds the free usage tier limits, you will be charged.

Launch a new intstance, click over to Community AMI’s and copy/paste the BitNami image in question:

Note the ebs based root device – now we’re cooking with crisco! After selecting the image, you’ll be guided through a fairly simple setup. The free instance is “Micro”, so be sure to change the default selection:

You should definitely spend a few minutes considering the region in which to launch the instance. If you’re using Analytics, checkout your Map Overlay to see from where your visitors are coming. Our biggest block primarily comes from the U.S. West Coast, but there are significant folks throughout the continental U.S. and Europe.

We ultimately went with the mid-point “U.S. East” region.

Do not take the default security group setting as it provides neither SSH nor HTTP access. Create your own custom group adding at least these ports.

After your instance launches, click on it in the AWS Management Console. The description tab below will list a “Public DNS”. Point your browser to this URL and you’ll see the BitNami Welcome page with a few links including your new WordPress install. Not too shabby – a running blog on a new virtual server in all in the space of about 15 minutes.

At some point creation phase above, you were prompted to download a .pem file – this is your SSH keypair to get in the back door of your new instance. You’ll need to chmod 644 my_aws_ec2.pem & then ssh -i my_aws_ec2.pem bitnami@public_dns.amazonaws.com to access your EC2 server via ssh. The bitnami user has sudo access.

Last, but not least, make an EBS snapshot of your running instance so you have a known fallback before entering the next phase of the migration. Don’t forget about this capability – at every step where you feel you’ve accomplished an important step, back it up!

Migrating Your WordPress Blog

On your existing blog, dump your database and tarball your WordPress htdocs path.

$ mysqldump -uwordpress -p awo > awo_wp.sql
$ tar -cjf awo_htdocs.bz2 /home/awo/public

Now, let’s get those dumps copied over to your new EC2 instance:

$ scp -i my_aws_ec2.pem awo_wp.sql awo_htdocs.bz2 bitnami@public_dns.amazonaws.com:/home/bitnami/originals/

And, don’t forget to copy over any releveant configurations and performance tweaks in your Apache (apache2.conf, httpd.conf) and MySQL (my.cnf, conf.d/*.cnf) setups.

Moving on to the IP address. Goto the Elastic IP’s entry in your AWS console. Allocate New Address and associate it with your newly created (and probably only) EC2 instance. While we’re waiting for the DNS to catch up, edit your local (meaning the computer you’re using to do the migration) /etc/hosts accordingly, overriding your blog domain with this new IP address.

$ sudo vi /etc/hosts
...
<NEW_ELASTIC_IP> www.agileweboperations.com

On the new instance do something like the following (remembering your EBS snapshot above ),

$ rm -Rf /opt/bitnami/apps/wordpress/htdocs/
$ tar -xjf ~/originals/awo_htdocs.bz2 /opt/bitnami/apps/wordpress
$ mysql -uroot -pbitnami bitnami_wordpress < ~/originals/awo.sql

Note the location of the BitNami wordpress, and the MySQL root password and wordpress database schema.

Now, let’s setup a virtual host to point to your new blog:

$ sudo vi /opt/bitnami/apache2/conf/httpd.conf
...
# Virtual hosts
Include conf/extra/httpd-vhosts.conf
...
$ sudo vi /opt/bitnami/apache2/conf/extra/httpd-vhosts.conf
...
<VirtualHost *:80>
    DocumentRoot "/opt/bitnami/apps/wordpress/htdocs"
    ServerName agileweboperations.com 
    ServerAlias www.agileweboperations.com
    ErrorLog "logs/awo-error_log"
    CustomLog "logs/awo-access_log" common
</VirtualHost>
...

Any special apache configuration (KeepAliveTimeout, ServerTokens, etc.) can be edited @ /opt/bitnami/apache2/conf/extra/httpd-default.conf. Be sure to include these edited files in the base apache conf:

$ sudo vi /opt/bitnami/apache2/conf/httpd.conf
...
# Server-pool management (MPM specific)
Include conf/extra/httpd-mpm.conf
...
# Virtual hosts
Include conf/extra/httpd-vhosts.conf
...
# Various default settings
Include conf/extra/httpd-default.conf

Finally, remember your canonical redirects! No duplicate content here:

$ vi /opt/bitnami/apps/wordpress/htdocs/.htaccess
RewriteEngine On
RewriteCond %{HTTP_HOST} .
RewriteCond %{HTTP_HOST} !^www\.agileweboperations\.com
RewriteRule (.*) http://www.agileweboperations.com/$1 [R=301,L]

Restarting processes with Bitnami is accomplished using the ctlscript.sh. To restart Apache, just run sudo /opt/bitnami/ctlscript.sh restart apache.

MySQL my.cnf may be edited @ /opt/bitnami/mysql/my.cnf and the restart is accomplished in much the same way : sudo /opt/bitnami/ctlscript.sh restart mysql.

You can also use ctlscript.sh to check out your apache & mysql status:

$ sudo /opt/bitnami/ctlscript.sh status
apache already running
mysql already running

Gotchas & Caveats

Installing ec2-api-tools is not possible on a t1.micro Ubuntu instance. If you know you’ll need this, either install the instance initially as m1.small or you can stop/switch an existing instance (which is what I had to do).

This was also a bit of a learning experience, as stopping an instance removes the elastic IP address. Sure, makes sense – but if you don’t know this you’ll wonder why you can no longer ssh into your restarted instance. Additionally the “Public DNS” name will also dynamically change upon starting an instance. Plus, just starting an instance immediately costs you 1 hour of computing time, so don’t be surprised to see a bit more usage than you’d expect if you do this a few times.

YMMV, but the bitnami:daemon user:group which owned /opt/bitnami/apps/wordpress/htdocs was not sufficient for performing any “inline” wordpress upgrades/installs – meaning trying to upgrade/install a plugin asked for FTP access information. Performing a sudo chown -Rf daemon:daemon /opt/bitnami/apps/wordpress/htdocs did the trick, however. Anyone more familiar with BitNami, please feel free to step in and clarify.

MySQL died the first night after relaunch. A strange “Killed” showed up in the /opt/bitnami/mysql/data/mysqld.log:

101229 15:41:33 [Note] /opt/bitnami/mysql/bin/mysqld.bin: ready for connections.
Version: '5.1.50'  socket: '/opt/bitnami/mysql/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)
Killed
101230 00:15:51 mysqld_safe Number of processes running now: 0
101230 00:15:51 mysqld_safe mysqld restarted
101230  0:15:53 [Note] Plugin 'FEDERATED' is disabled.
Killed
101230 00:15:54 mysqld_safe mysqld from pid file /opt/bitnami/mysql/data/ip-10-112-91-108.pid ended
101230 07:32:02 mysqld_safe Starting mysqld.bin daemon with databases from /opt/bitnami/mysql/data
...

Turns out, the Bitnami image makes use of Linux’s OOM Killer which takes a heavy-handed approach to potential Out of Memory issues before they over swap the server. The problem was I forgot to include the httpd-mpm.conf file I had altered so carefully above in the httpd.conf, so there were a bunch more simultaneous clients than this micro server could manage.

After upgrading to WP 3.0.4, and doing a sudo apt-get update & sudo apt-get upgrade, accessing any pages in wp-admin somehow deadlocked the entire server (loads of 10-15 observed). The only thing that finally fixed this was restarting the entire instance.

After you get an elastic IP setup, delete the /opt/bitnami/updateip file. Otherwise, restarting the instance, Bitnami resets the wp_options values to the Public DNS server name. If you forget, you must change this back or else your blog won’t work:

$ mysql -uroot -p bitnami_wordpress
mysql> update wp_options set option_value = 'http://www.agileweboperations.com' where option_name in ('siteurl', 'home');
Query OK, 2 rows affected (0.00 sec)

Let’s see how are “free” year works out – for such little effort (~1 day), it’s well worth the savings (and maybe AWO gets into the black for the first time in 2011 ). And, check out where the rubber meets the road – our Pingdom response times before and after the relaunch:

Professional Grade Virtual Computing for “Free”

In case you missed my other posts on acquiring, maintaining, backing-up and monitoring your virtual server, here’s the complete list:
• Performance : WordPress Caching For High Performance EC2 Micro Instances
• Backups : Poor Man’s Automated Snapshots for EC2
• Monitoring : Cloudkick Monitoring for EC2

Other posts:

• WordPress Caching For High Performance EC2 Micro Instances
• Poor Man’s Automated Snapshots for EC2
• Free Cloudkick Monitoring for EC2

2010 was a great year here at Agile Web Development & Operations. Since 2008, we’ve written close to 200 posts (many not so good, but, luckily, there were also quite a few you liked). You guys left over 450 comments (You Rock!) and spammers tried to swamp us with over 22.000 comments (they Suck! and that’s the reason why we still have comment moderation on ).

So many more folks were interested in what we (and our great guest posters – more about that later) wrote that our traffic (over 84.000 visitors) and our subscriber numbers (over a thousand already) more than doubled through 2010 – wow!

Here are some of our most interesting posts (as well as some big mistakes) from 2010. Enjoy!

Our detailed tutorial about how to make Xen run on Debian Lenny Dom0 with Ubuntu Lucid Guest was definitely one of the winners in 2010. It was viewed close to 7k times and received over 40 comments.

Another clear hit was our list of 20 Devops Guys you should follow. If you haven’t checked it out yet, do so. Now. These guys are awesome!

Other popular posts were:

• Using Kanban for DevOps projects
• Launch Dates: The Good, the Bad, and the Ugly
• Visualizing Small Batch Sizes with git
• Bugfixes without tests are anti-fixes

And, last, but not least (on the contrary!), we had a great bunch of guest posters – especially the DevOps series we started in November 2010 was fantastic. Thanks again to everybody who participated. And, even cooler, it’s not over yet. Make sure you subscribe not to miss some great upcoming posts.

And, yes, we promised to tell you our worst stuff in 2010. Here you go:

I totally blew that one: Waterfall, SCRUM and Lean Software Development simulation as teaching platform. I tried to teach my students the benefits of agile by simulating a waterfall project first. Well, they rebelled and I stopped the simulation within the first hour. No student in the world can survive one day of a waterfall project it seems.

Talk about reinventing the wheel: Simple Database Updates with Ruby. I suffered with this for almost 6 months before waking up and using rake db:migrate. Ouch!

We hope you’ve had a good start in 2011 and we’re looking forward to bringing more interesting stuff to you!

Dan and Matthias

No related posts.

For nearly two years Dan and I have shared our experiences and ideas about agile development and system administration. With every post we hoped to be helpful, and maybe some of them even were…

Now, as we approach 500 subscribers, we would like to ask you, our dear readers, how we could help you to become even more agile and have more fun doing your job.

Please take our short survey at: http://www.surveymonkey.com/s/LJVMRG9 and tell us about your situation, your goals and how you would profit most from our experiences.

We’d love to hear from you!

Dan & Matthias

Other posts:

• 1st Birthday: Best of Agile Web Operations
• About Agile Web Development & Operations
• The 12 principles behind the agile manifesto adapted to web operations

It’s time to say “Thank you” to all of you, our loyal readers. Thanks for reading, commenting and subscribing to our posts for exactly one year now. That’s right, Agile Web Operations is now one year old. I can hardly put into words how great it is to have you with us!

103 posts and 113 comments. That’s the result of our first year. And, exactly 200 of you have subscribed to our RSS feed – an excellent choice. For all of you who joined recently (and to remind you veterans), check out the five most read posts:

1. Puppet or Capistrano – Use the Right Tool for the Job (Guest post by Andrew Shafer)
2. Puppet vs. Capistrano – a short comparison
3. System Configurations + Code Revisions = Continuous Integration FTW (Guest post by Patrick Debois)
4. Thoughtworks Mingle vs. Pivotal Labs Tracker
5. Agile Tool Vendors: Please don’t try to manage complexity – simplify my life!

And don’t miss out on our three most commented posts:

• Agile on steroids
• Configuration Management remixed: Introducing Carpet
• Visible Ops: Rolling Out Change Management

I hope you enjoy reading Agile Web Operations as much as we do writing it. Keep the comments flowing and don’t forget to tell your friends!

We’re looking forward to the coming years of Agile Web Operations.

Sincerely yours,

Dan & Matthias

Other posts:

• Agile Web Operations: What do YOU want it to be?
• About Agile Web Development & Operations
• Improving Operations with Agile