Continuously Deploying Your Chef Cookbooks In Travis

Stuart Harrison

Here in the ODI Tech Team, we love robots (heck, we even have a Head of Robots), and, as a result, all our code is deployed by robots (using Chef).

As has been mentioned before, our workflow is very much based on Github flow. We write some code (with tests), publish it to a branch, and then open a pull request. If the tests pass (on Travis), someone else reviews the code and merges it. Once the tests pass on the master branch, a tag is applied to the code and the Chef robots (which run every 10 minutes or so), pick the code up and deploy it to our servers.

Up until now, we haven’t been able to apply this approach to our Chef cookbooks. We test them, sure, but it’s tricky to test infrastructure on Travis, because it’s difficult (nay, impossble) to spin up virtual machines within the virtual machines that Travis spin up to test our code.

However, we’ve recently started using Test Kitchen to test our Chef recipes, and thanks to some wonderful tools that other people have built, this is a lot easier.

When running Test Kitchen on our local machines, the default behaviour is to use Vagrant to launch a virtual machine, apply the recipes and then use ServerSpec to test the behaviour (For more information, check out the Test Kitchen Docs).

When we are running our tests on Travis, however, we need to do things a little differently. Instead of spinning up a virtual machine on Travis, we use Kitchen Rackspace to spin up a VM on Rackspace for us. The tests are then run on that machine, and the server is destroyed for us, meaning we pay pennies for a short-lived server.

Getting this set up is a little tricky, so I thought I’d give something back and document this process, so that future generations (that’s you, dear reader) might benefit.

Assuming you’ve got Test Kitchen set up, and have everything set up in Travis, the first thing to do is add kitchen-rackspace to your Gemfile like so:

gem 'kitchen-rackspace'

Then run bundle install. The next thing to do is create a new .kitchen.yml file. (We’ve called it .kitchen.cloud.yml as this seems to be the convention), and add the following:

---
driver:
  name: rackspace
  rackspace_username: <%= ENV['RACKSPACE_USERNAME'] %>
  rackspace_api_key: <%= ENV['RACKSPACE_API_KEY'] %>
  rackspace_region: lon
  require_chef_omnibus: latest

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-12.04

Then add your run list to the file from your existing .kitchen.yml file. You may want to tweak some of the settings above to suit your needs. See the kitchen-rackspace docs for more.

It’s also worth pointing out that the platforms section will only work if the name is referred to in the JSON file included in the kitchen-rackspace gem. If you want to use a different platform, you can list the ones available to you by running:

 knife rackspace image list --rackspace-region=lon --rackspace-username=$RACKSPACE_USERNAME --rackspace-api-key=$RACKSPACE_API_KEY

Where $RACKSPACE_USERNAME is your Rackspace Username and $RACKSPACE_API_KEY is your Rackspace API key.

You’ll notice we’re passing some environment variables to the YAML file. These can be encrypted and added to your .travis.yml file on the command line like so:

travis encrypt RACKSPACE_USERNAME=YOUR_RACKSPACE_USERNAME --add
travis encrypt RACKSPACE_API_KEY=YOUR_RACKSPACE_API_KEY --add

Once that’s done, you need to add a few more things to your .travis.yml file. Firstly, you need to make sure the Travis VM has an ssh key (they don’t by default). This allows kitchen-rackspace to transfer the necessary files to your Rackspace node, trigger the Chef run, and run the tests:

before_script:
   - ssh-keygen -f ~/.ssh/id_rsa -t rsa -N ''

The next thing to do is add the environment variable KITCHEN_YAML to your Travis env to make sure Test Kitchen knows what YAML file to use. The env section of your .travis.yml file should look something like this:

env:
  global:
  - secure: SOME_ENCRYPTED_TOKEN
  - secure: ANOTHER_ENCRYPTED_TOKEN
  - KITCHEN_YAML=.kitchen.cloud.yml

Then, the next thing to do, is tell Travis what command to run to run your tests. In our live projects, we’ve got a Rakefile which runs some other tests as well as Test Kitchen, but, in the interests of keeping it simple, we’ll just run Test Kitchen here:

script: 
  - travis_wait 35 kitchen test --destroy=always

The first thing to note is the kitchen test command is preceeded by travis_wait 35, this is a special Travis command which stops Travis from timing out until the following command has been running for 35 minutes. Normally Travis will time out after 10 minutes, and this often isn’t enough for the whole test run to happen. We’re building a virtual machine from scratch remember? You can see more about travis_wait here

The --destroy=always tag is important because by default, Test Kitchen only destroys your box after a successful test. We don’t want additional servers hanging around and costing us money, so we’ll kill them every time.

By now, your .travis.yml file should look something like this:

language: ruby
rvm:
- 2.1.0
before_script:
  - ssh-keygen -f ~/.ssh/id_rsa -t rsa -N ''
script:
  - kitchen test --destroy=always
env:
  global:
  - secure: SOME_ENCRYPTED_TOKEN
  - secure: ANOTHER_ENCRYPTED_TOKEN
  - KITCHEN_YAML=.kitchen.cloud.yml

You could just stop there, add your project to Travis and push your changes, then upload your cookbook to the Chef server, but as I mentioned before, we’re all about continuous deployment here, so we want to go one step further - if the tests pass on master, we want to upload our cookbook to the Chef server.

To get this done, you’ll need to make sure you’re using Berkshelf. Firstly, add:

gem 'berkshelf'

To your Gemfile, then run:

bundle install
berks init .

You will then have a Berksfile in your project root. You can use this to manage cookbook dependencies, more on this on the Berkshelf website here, but for the sake of argument, we’ll just leave the Berksfile as it is.

The next step is to add a berkshelf.json file to your repository. We tend to put this in a deploy subdirectory.

{
  "chef": {
    "chef_server_url": "https://chef.theodi.org",
    "client_key": "deploy/key.pem",
    "node_name": "odi"
  },
  "ssl": {
    "verify": false
  }
}

This tells Berkshelf where to upload the cookbook to. Note it needs the client key PEM file. This should not be added to version control. In order to get this, we add an encrypted version to git, and decrypt it on Travis with an environment variable.

export CHEF_KEY=SOME-UNIQUE-KEY
openssl aes-256-cbc -k "$CHEF_KEY" -in deploy/key.pem -out deploy/key.enc -a -e

We then need to add our unique key to Travis like so:

travis encrypt CHEF_KEY=${CHEF_KEY} --add

Then add deploy/key.enc to version control.

For convenience, we usually add a rake task to handle the berkshelf upload:

namespace :berkshelf do

  desc "Upload cookbook to chef server"
  task :upload do
    sh "bundle exec berks upload -c deploy/berkshelf.json"
  end

end

Now, in your travis config, after successful master builds, you want to decrypt the PEM file and run the rask task:

after_success:
- openssl enc -d -aes-256-cbc -k $CHEF_KEY -in deploy/key.enc -out deploy/key.pem
- chmod 600 deploy/key.pem
- bundle exec berks install
- "[ \"$TRAVIS_BRANCH\" == \"master\" ] && [ \"$TRAVIS_PULL_REQUEST\" == \"false\"] && bundle exec rake berkshelf:upload"

Now, when your build passes, Travis should try to upload the new cookbook to the Chef server. Bingo!