Learn How to orchestrate Your Infrastructure Fleet with Chef Provisioning

Chef Provisioning is a relatively new member in the Chef family. It can be used to build infrastructure topologies using the new machine resource. This blog post shows how this is done.

You bring up and configure individual nodes with Chef all the time. Your standard workflow would be to bootstrap a node, register the node to a Chef server, and then run Chef client to install software and configure the node. You would rinse and repeat this step for every node that you want in your fleet. Maybe you have written a nice wrapper over Chef and Knife to manage your clusters using Chef. Until recently, Chef did not have any way to understand the concept of cluster or fleet.

So if you were running a web application with some decent traffic,there would be a bunch of cookbooks and recipes to install and configure: web servers, DB server, background processor, load balancer, etc. Sometimes, you might have additional nodes for Redis or RabbitMQ. So let us say, your cluster consists of three web servers, one DB server, one server that does all the background processing, like generate PDFs or send emails etc., and one load balancer for the three web servers. Now if you wanted to bring such a cluster for multiple environments, say “testing”, “staging,” and “production,” you would have to repeat the steps for each environment; not to mention, your environments could possibly be powered by different providers–production and staging on AWS, Azure, etc. But testing could possible be on local infrastructure, maybe in VMs. This is not difficult, but it definitely makes you wonder whether you could do it better–if only you could describe your infrastructure as code that comes up with just one command.
That is exactly what Chef Provisioning does. Chef Provisioning was introduced in Chef version 12. This helps you describe your cluster as code and build it at will as many times as you want and on various types of clouds, virtual machines, or even on bare metal.

The Concepts

Chef provisioning depends on two main pillars–machine resource and drivers.

Machine Resource

“machine” is an abstract concept of a node from your infrastructure topology. It could be an AWS EC2 instance or a node on some other cloud provider. It could be a Vagrant-based virtual machine, a Linux container, or a Docker instance. It could even be a real, physical bare-metal machine. “machine” and other related resources (like machine_batch, machine_image, etc.,) can be used to describe your cluster infrastructure.
Each “machine” resource describes whatever it does using standard Chef recipes. General convention is to describe your fleet and its topologies using “machine” and other resources in a separate file. We will see this in detail soon, but for now here is

#setup-cluster.rb
	machine 'server' do
	   recipe 'nginx'
	end
	machine 'db' do
	   recipe 'mysql'
	end

$ gem install chef-provisioning-aws
$ export CHEF_DRIVER=aws

driver 'fog:AWS'

require 'chef/provisioning/aws_driver'
with_driver 'aws'
machine 'server' do
   recipe ‘web-server-app'
end

machine 'server' do
   driver 'vagrant:/opt/vagrantfiles'
   recipe 'web-server-app'
end

#recipes/webapp.rb
require 'chef/provisioning'
 machine 'db' do
   recipe ‘database-server’
end
machine 'webapp' do
   recipe 'web-app-stack'
end

sudo CHEF_DRIVER=aws chef-client -r "recipe["cluster::webapp"]"

1.upto(5) do |i|
   machine "webapp-#{i}" do
      recipe 'web-app-stack'
   end
end

machine 'webapp' do
   action :setup
end
machine 'db' do
   action :destroy
end

machine 'webapp' do
   action :setup
end
remote_file 'somefile.tar.gz' do
   url 'https://example.com/somefile.tar.gz'
end
machine 'db' do
   action :setup
end

machine_batch do
   action :setup
   machines 'web-app-stack', 'db'

machine_image 'web_stack_image' do
   recipe ‘web-app-stack’
end

machine 'web-app-stack' do
   from_image 'web_stack_image'
end

1.upto(2) do |node_id|
  machine “web-app-stack-#{node_id}”
end
load_balancer "web-app-load-balancer" do
   machines %w(web-app-stack-1 web-app-stack-2)
end

with_machine_options :ssh_username => '',
  :bootstrap_options => {
    :key_name => '',
    :image_id => ‘',
    :instance_type => ‘’,
    :security_group_ids  => ''
  }