Amazon EC2 Instances With Opscode Chef Using Knife

by on March 29, 2011 · 22 comments

One of the nice features of opscode chef is it’s integration with cloud providers like amazon EC2. Knife, opscode chef’s command line client tool, makes it possible to create and bootstrap a VM in just one line – if you go through a few setup steps. In this article, I want to show you how to setup your chef installation and AWS configuration so you can easily bootstrap new VMs with chef.

Prepare SSH Access To Your Amazon EC2 Instances

Configure Your Amazon Security Group
As Amazon blocks all incoming traffic to your EC2 instances by default. you’ll need to open the SSH port for knife to access a newly created instance. This is pretty easy. Just login to the AWS console and navigate to the EC2 tab. Go to Security Groups and add a rule for SSH with Source 0.0.0.0/0. This enables SSH connections from anywhere in the world. If you want to limit access to just your home or work network, enter the corresponding IP address instead of 0.0.0.0/0.

Generate Key Pair in AWS Console
To enable API access to your AWS account you need to create a key pair. knife will use it to create, list, and destroy VMs.

Just select Key Pairs in your AWS console’s EC2 tab and press Create Key Pair. Give it a name (e.g. knife so you know that this key pair will be used by knife) and store the downloaded private key knife.pem in ~/.ssh/knife.pem

Prepare your ~/.ssh/config to avoid host key mismatch errors
Open your ~/.ssh/config and add:

Host ec2*compute-1.amazonaws.com
  StrictHostKeyChecking no
  User ubuntu
  IdentityFile  /Users/mm/.ssh/knife.pem

(make sure you fix the path to your home dir)

Now, SSH will work. Time to move on to the next step.

Configure knife Enabling it to Manage EC2 Instances

Tell knife about your AWS credentials
Put AWS credentials (access key and secret key found in your AWS profile) into ~/.chef/knife.rb like this

knife[:aws_access_key_id] = "..."
knife[:aws_secret_access_key] = "........."

Choose an AMI for your EC2 instances
If you run small instances, you’re bound to 32-bit only so make sure you choose a 32-bit AMI as well.

Note: the chef ‘ubuntu10.04-dpkg-ree’ bootstrap template will NOT work with small 32-bit instances as it is 64-bit.

Look here for Ubuntu 10.04 LTS AMI IDs

Create the EC2 instance using opscode chef knife
Now, it’s time to use knife to fire up and configure a new EC2 instance.

$ knife ec2 server create -r "role[ubuntu]" -I ami-399ca94d -f m1.small -S knife -i ~/.ssh/knife.pem --ssh-user ubuntu --region eu-west-1 -Z eu-west-1a
  • "role[ubuntu]" is the run_list I want to associate with the newly created node. You can put any roles and recipes you like here
  • -I is the AMI ID you selected earlier
  • -f is the Amazon EC2 instance type (see API name)
  • -S is the name you gave to the EC2 key pair generated in the AWS console
  • -i points to the private key file of that EC2 key pair as downloaded when the keypair was created in the AWS console
  • --ssh-user the official Ubuntu EC2 AMIs use ubuntu as the default user
  • --region eu-west-1 If you want your instances to be deployed in any specific Amazon AWS region, add this parameter and the desired region
  • -Z eu-west-1a is the availability zone within your region (i.e. you have an existing disk volume you need to made availble to this instance)

ATTENTION: make sure to kill the instance again if not needed anymore ;-)

Managing EC2 Instances With knife

Once you’ve started up at least one instance with knife, you can use it to find running EC2 instances like this:

$ knife ec2 server list --region eu-west-1

(make sure you use the correct --region parameter)

And, if you want to get rid of an instance (terminate instance and delete chef node), it’s as easy as:

$ knife ec2 server delete i-XXXXXXXX --region eu-west-1
$ knife node delete i-XXXXXXXX

(i-XXXXXXXX is the ID of the instance as found in the AWS console or a knife ec2 server list call)

After getting the initial setup right it’s a breeze to start, list, and stop Amaon EC2 instances with opscode chef knife. With just a single command you can instantiate a new server, bootstrap it as a chef-client and run all chef recipes defined in the run_list. Pretty sweet. What are your experiences with knife and EC2 (or other cloud providers)? Let us know in the comments…

Did you enjoy this article? Get new articles for free by email:


Comments

  1. Sean Gallagher says

    Thanks for the post!
    Check your command for:
    “Create the EC2 instance using opscode chef knife”
    >> should be -I (capital) for ami and -i (lowercase) for pem file

  2. Rajat says

    Hi Matthias

    Thanks for the post. I have a ques. I want to use my own custom ami to launch the ec2 machine. How to do that?

    Thanks in advance
    Rajat

  3. anoldguy says

    I think you’re missing a ‘-r’ before “role[ubuntu]” in your knife command. It should look like this:

    knife ec2 server create -r “role[ubuntu]” -I ami-399ca94d -f m1.small -S knife -i ~/.ssh/knife.pem –ssh-user ubuntu –region eu-west-1 -Z eu-west-1a

  4. EJC says

    No matter what I do, I keep getting the message:

    ERROR: You have not provided a valid image (AMI) value. Please note the short option for this value recently changed from ‘-i’ to ‘-I’.

    I even tried the exact example you guys gave as well as with some of the ami’s I’ve created with no luck. Any suggestions? Adding -VV doesn’t help either…

  5. EJC says

    hmmmm- interesting – but I’m not using any double dashes anywhere. The help output suggests -I = the AMI for the server and the -i = the identitiy file – so I’m NOT mixing those up either.

    Any other suggestions?

    • EJC says

      Ok, started over, created a new ami and left it running. From there I took note of zone and region. Then I could run the command with the region and zone options and now it’s sitting waiting for sshd…

      I get the feeling it’s going to crash out…

  6. EJC says

    Yeah, not sure what else to do, but the ssh portion never seems to connect/get set up properly.

    Any final suggestions?

  7. EJC says

    ok, got it – enabled the ssh rule for the default security group (doh!).

    Next issue is we’re hosting our own chef server – this writeup is ONLY good for people NOT hosting their own chef server?

  8. Ran says

    Hi, thanks for the post, this was very useful!

    But I’m having trouble, and google didn’t help…
    I get the following error:
    ERROR: Fog::Compute::AWS::Error: SignatureDoesNotMatch => The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

    I get this error for all knife ec2 commands (server create, server list etc)
    But when using the same key ID and secret access key with the EC2 command line tools provided by amazon, all is well! So I’m a bit confused…
    I tried two different amz accounts and the same problem.
    I tried using -A and -K explicitly, didn’t work. I tried putting them in ~/.chef/knife.rb file and still no dice.

    I haven’t found anyone else with the same error on google. Any idea how to proceed?

    $ knife -v
    Chef: 10.14.2

    thank you!

  9. says

    Just the tutorial I needed. Thanks for the great writeup! One small issue I run into (it doesn’t effect the creation/deletion of instances but was curious if you had a solution) is that I get errors as chef tries to create a node in the chef server. I’m using chef solo so I was wondering if there was a setting I’m missing to prevent knife from trying to create/remove the nodes on the server. Any insight is appreciated. Thanks again for the good writeup!

  10. Chris Olido says

    Ih having problems with the validation.pem. Technically my validation.pem is correct and identical to the chef server and to the client when i compare it.

    When im issuing:

    knife ec2 server create -r “role[wordpress]” -E staging -S www-sg -I ami-b2a2**** –flavor t1.micro –region ap-southeast-1 -G public_www -x ubuntu -N testme

    im having an error like this:

    Chef::Exceptions::InvalidPrivateKey: The file /etc/chef/validation.pem does not contain a correctly formatted private key.
    The key file should begin with ‘—–BEGIN RSA PRIVATE KEY—–’ and end with ‘—–END RSA PRIVATE KEY—–’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/rest/auth_credentials.rb:68:in `rescue in load_signing_key’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/rest/auth_credentials.rb:60:in `load_signing_key’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/rest/auth_credentials.rb:34:in `initialize’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/rest.rb:64:in `new’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/rest.rb:64:in `initialize’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/client.rb:325:in `new’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/client.rb:325:in `register’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/client.rb:407:in `do_run’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/client.rb:176:in `run’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/application.rb:140:in `run_chef_client’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/application/client.rb:274:in `block in run_application’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/application/client.rb:267:in `loop’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/application/client.rb:267:in `run_application’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/application.rb:72:in `run’
    /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/bin/chef-client:26:in `’
    /usr/bin/chef-client:23:in `load’

    any recommendation would be appreciated.

    By the way Thanks again for the good writeup…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>