Managing Container Clusters with Terraform and Rancher

Infrastructure as code is a practice of codifying and automating the deployment and management of infrastructure with tooling. This allows for testing, reviewing, approving, and deploying infrastructure changes with the same processes and tools as application code. In this blog post, we’ll walk through using Rancher and Terraformto implement infrastructure as code, using the recently built-in Rancher Terraform provider.

Terraform from Hashicorp is a tool for abstracting service and provider APIs into declarative configuration files. It then tracks the state of the infrastructure and converges it to match the specified configuration. Terraform ships with built-in support for a variety of cloud providers (AWS, CenturyLink Cloud, Google Cloud, Microsoft Azure, OpenStack, VMware vSphere, etc.) and other services such as BitBucket, GitHub, Fastly, Heroku DNSimple, and Rancher. The full list of providers can be found at online in the Terraform docs.

2a6ca22242b1fde0341dd182d195f70cdc1759b9 Free eBook: Compare architecture, usability, and feature sets for Kubernetes, Mesos/Marathon, and Docker Swarm Beginning with version 0.8.0, Terraform is able to manage resources within a Rancher cluster using the built-in provider. The full list of supported resources for Rancher can be found here. Any issues should be reported at Github.

Setting up

This post assumes that a Rancher server is available for testing, and the latest version of Terraform is installed. If a Rancher server is not available, one can be started locally using the following Docker command:

$ docker run -d -p 8080:8080 rancher/server:v1.3.3

Configuring the Rancher provider

The first step to managing a Rancher server with Terraform is to configure the rancher provider for Terraform. Create a working directory for this example and place the following code in the rancher.tf within the directory:

provider "rancher" {
  api_url = "http://localhost:8080" //
  access_key = "" //
  secret_key = ""
}

The only required configuration for the rancher provider is api_key. This should be configured to point to the Rancher server location. Optionally, the API access and secret keys can be configured with the access_key and secret_key properties as shown. If Rancher server does not have access control configured, then these settings can be omitted. If access control is enabled, then API credentials are required. Credentials can be created by following the instructions here. Rancher is also able to access these properties using the environment variables RANCHER_URL, RANCHER_ACCESS_KEY, and RANCHER_SECRET_KEY respectively. For security purposes, it is advisable to use the environment variables or Terraform variables to provide credentials.

Creating environments

Now that the provider is configured, Terraform can create new resources. Start by creating a new Environment within Rancher by placing the following code in the demo.tf file in the working directory.

resource "rancher_environment" "demo" {
  name = "blog-demo"
  description = "Demonstration environment"
  orchestration = "cattle"
}

First, a new resource is declared with the type of rancher_environment and a logical name of demo. This is used for referencing this resource within the Terraform configuration. The name property is required for the environment and specifies the name of the environment in Rancher. Optionally, a description can be provided. Finally, the orchestration engine to use for the environment is specified. This value can be cattle, kubernetes, swarm, or mesos. If omitted, the value will default to cattle. Once specified, Terraform can dry-run and apply the configurations by executing terraform plan and terraform apply respectively:

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage.
+ rancher_environment.demo
    description:   "Demonstration environment"
    name:          "demo"
    orchestration: "cattle"

Plan: 1 to add, 0 to change, 0 to destroy.
$ terraform apply
rancher_environment.demo: Creating...
  description:   "" => "Demonstration environment"
  name:          "" => "demo"
  orchestration: "" => "cattle"
rancher_environment.demo: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path below. This state is required to modify and destroy your infrastructure, so keep it safe. To inspect the complete state use the terraform show command.

State path: terraform.tfstate

Getting host configuration information

Terraform can create the registration token information for a host to join and environment using the rancher_registration_token resource. Add the following to the demo.tf file:

resource "rancher_registration_token" "demo-token" {
  environment_id = "${rancher_environment.demo.id}"
  name = "demo-token"
  description = "Host registration token for Demo environment"
}

The environment is declared by creating a new rancher_registration_token resource with the logical name of demo-token. The token is declared to be created in the demo environment by populating the environment_id property with the id property from the previously created environment. This uses Terraform’s built in interpolation. and resource dependency system. Finally, the name is a required property and an optional description can be provided. Run terraform plan and terraform apply to view and apply the changes once again. Note that any IDs in the sample output may be different.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

rancher_environment.demo: Refreshing state... (ID: 1a7)
...

+ rancher_registration_token.demo-token
    command:          ""
    description:      "Host registration token for Demo environment"
    environment_id:   "1a7"
    name:             "demo-token"
    registration_url: ""
    token:            ""


Plan: 1 to add, 0 to change, 0 to destroy.
$ terraform apply
rancher_environment.demo: Refreshing state... (ID: 1a7)
rancher_registration_token.demo-token: Creating...
  command:          "" => ""
  description:      "" => "Host registration token for Demo environment"
  environment_id:   "" => "1a7"
  name:             "" => "demo-token"
  registration_url: "" => ""
  token:            "" => ""
rancher_registration_token.demo-token: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

The rancher_registration_token resource outputs a few useful properties. First, it captures the entire command value. This is the value that is visible in the Rancher UI when adding a Custom Host. It contains the entire command for staring the Rancher agent on a machine with Docker to join this environment. Additionally, the registration_url and token properties are captured. These contain the full URL for the Rancher agent to connect to as specified in the command property and the environment specific path token from that URL respectively. To view the host command, an output can be configured in Terraform to capture the value and write it to the console. Add the following block to the demo.tf file.

output "rancher_agent_command" {
  value = "${rancher_registration_token.demo-token.command}"
}

Now run a terraform apply to configure the output value:

$ terraform apply
rancher_environment.demo: Refreshing state... (ID: 1a7)
rancher_registration_token.demo-token: Refreshing state... (ID: 1c2)

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

rancher_agent_command = sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.1.3 http://localhost:8080/v1/scripts/A17BDBB62228B4B18B4A:1485619200000:undsWydVkrYSHqd2NNU5fVDdx7s

The rancher_registration_token outputs can be used to configure the launch parameters for other resources. For example, if Amazon Web Services is being used to create an autoscaling group of instances to join the cluster, the registration command can be added to the user data like so:

resource "aws_launch_configuration" "demo-hosts" {
  name = "demo_hosts"
  image_id = "ami-6edd3078"
  instance_type = "t2.micro"
  root_block_device {
    volume_type = "gp2"
    volume_size = 8
    delete_on_termination = true
  }
  user_data = <
    permissions: "0755"
    content: |
      #!/bin/bash
      DOCKER_VERSION=$1
      apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
      mkdir -p /etc/apt/sources.list.d/
      echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" > /etc/apt/sources.list.d/docker.list
      apt-get update
      apt-get install -y linux-image-extra-virtual linux-image-extra-$(uname -r) docker-engine=$DOCKER_VERSION

  - path: /opt/rancher/rancher.sh
    permissions: "0755"
    content: |
      #!/bin/sh
      set -e
      umask 077

      ${rancher_registration_token.demo-token.command}

runcmd:
  - [ cloud-init-per, once, docker, /opt/docker/install.sh, "1.13.0-0~ubuntu-xenial" ]
  - [ cloud-init-per, once, rancher, /opt/rancher/rancher.sh ]
EOF
}

This block creates a launch configuration for an Ubuntu Xenial image in the us-east-1 region in AWS. It adds boot time scripts to install Docker, and to start the Rancher agent using the command string from the registration token. Notice that the command property is interpolated into the user data as part of a script file that is written out. This block is an example and the same pattern can be used for various operating systems or cloud providers.

Controlling Rancher Stacks with Terraform

Terraform can additionally create and manage stacks within a Rancher environment. The entire docker-compose.yml and rancher-compose.yml can be specified for the stack, or the Rancher catalog can be used to specify the entry and answers to create a stack. Add the following block to the demo.tf file to create a new stack from the community catalog entry for the Ghost blogging platform.

resource "rancher_stack" "ghost" {
  environment_id = "${rancher_environment.demo.id}"
  name = "ghost"
  description = "Ghost demo stack"
  catalog_id = "community:ghost:0"
  scope = "user"
  start_on_create = true
  environment {
    public_port = "80"
  }
}

The block adds a new rancher_stack resource with the logical name of ghost within the demo environment previously created. It specifies the name and description (optional) for the stack. It then configures the stack to be created from the entry named ghost in the community catalog using the version `0` of the template. These values can be discovered by utilizing the \“View in API\” feature of Rancher. Finally, it configures the answers to the questions defined in the catalog template using the environment block. Run terraform apply to create the stack in Rancher:

$ terraform apply
rancher_environment.demo: Refreshing state... (ID: 1a7)
rancher_registration_token.demo-token: Refreshing state... (ID: 1c2)
rancher_stack.ghost: Creating...
  catalog_id:               "" => "community:ghost:0"
  description:              "" => "Ghost demo stack"
  environment.%:            "" => "1"
  environment.public_port:  "" => "80"
  environment_id:           "" => "1a7"
  name:                     "" => "ghost"
  rendered_docker_compose:  "" => ""
  rendered_rancher_compose: "" => ""
  scope:                    "" => "user"
  start_on_create:          "" => "true"
rancher_stack.ghost: Creation complete
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Outputs:

rancher_agent_command = sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.1.3 http://localhost:8080/v1/scripts/A17BDBB62228B4B18B4A:1485622800000:E0L4yrjYajfxbg7mKSNxLZGup0

Open your Rancher dashboard, switch the demo environment, and note that the ghost stack is created and active:

Summary

Terraform can be used to codify infrastructure across many platforms including Rancher. Using a tool like Terraform allows for easily repeating configuration and setup of infrastructure. It also allows for incorporating infrastructure changes into standard development practices and processes such as versioning, reviews, and promotions.  

John Engelman (\@johnrengelman) is the Chief Technologist for devops and cloud activities at Object Partners(\@objectpartners) in Minneapolis, MN. He is an active contributor to the Terraform project and one of the authors of the Rancher provider. With over 120 employees in the Minneapolis, Omaha, and Chicago regions, Object Partners has been building custom software solutions for clients in the US and Europe since 1996. Learn more at objectpartners.com.