AWS and Rancher: Building a Resilient Stack

In my prior posts, I’ve written about how to ensure a highly resilient workloads using Docker, Rancher, and various open source tools. For this post, I will build on this prior knowledge, and to setup an AWS infrastructure for Rancher with some commonly used tools. If you check out the repository here, you should be able to follow along and setup the same infrastructure. The final output of our AWS infrastructure will look like the following picture: In case you missed the prior posts, they’re available on the Rancher blog and cover some reliability talking points. Lets use those learning and create a running stack.

Host VM Creation

The sections we will build are the three lower yellow section: Golden Image First, we will need a solution to create Docker hosts that use a reliable combination of storage drivers and OS. We would also like to replace these with different parts in the future. So we build our base VM, or the \“golden image\” as it is more commonly referred to. As for the tools, Packer will be used to communicate with the AWS API for creating VM images (and various other cloud providers). Ansible will be used to describe the provisioning steps in a readable manner. The full source can be found here, if you want to jump ahead. Since the previous chain of posts on reliability used Ubuntu 14.04, our example will provision a VM with Ubuntu 14.04 using AUFS3 for the Docker storage driver. To start, we create a Packer configuration called ubuntu_1404_aufs3.json. In this case, my config searches for the AMI ID for most recent 14.04 AMI ID on AWS us-east through source_ami_filter, which as of writing returns ami-af22d9b9. It also creates a 40GB drive attached as /dev/sdb, which we will use to store Docker data; We are using Docker 1.12.3, because it is supported in the latest Rancher’s compatibility matrix.

 {
 "variables": {
 "aws_access_key": "",
 "aws_secret_key": "",
 "docker_version": "1.12.4"
 },
 "builders": [{
 "type": "amazon-ebs",
 "access_key": "{{user `aws_access_key`}}",
 "secret_key": "{{user `aws_secret_key`}}",
 "region": "us-east-1",
 "source_ami_filter": {
 "filters": {
 "virtualization-type": "hvm",
 "name": "*ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*",
 "root-device-type": "ebs"
 },
 "most_recent": true
 },
 "ami_virtualization_type": "hvm",
 "instance_type": "m3.medium",
 "ssh_username": "ubuntu",
 "ami_name": "Docker {{user `docker_version`}} Ubuntu 14.04 AUFS3 {{timestamp}}",
 "launch_block_device_mappings": [{
 "device_name": "/dev/sdb",
 "volume_size": 40,
 "volume_type": "gp2",
 "delete_on_termination": true
 }],
 "tags": {
 "OS_Version": "Ubuntu",
 "Release": "14.04",
 "StorageDriver": "AUFS3",
 "Docker_Version": "{{user `docker_version`}}"
 }
 }]
 }

 $> packer validate ubuntu_1404_aufs3.json
 Template validated successfully.

Great! It passes validation, but if we actually ran it, Packer would just create a copy of the base AMI with a 40GB drive attached, which isn’t very helpful. To make it useful, we will also need to provision Docker on it. Packer has built-in hooks for various configuration management (CM) tools such as Ansible, Chef, and Puppet. In our case, we will use the Ansible provisioner.

{
"variables": ["..."],
"builders": ["..."],

"provisioners": [
{
"type": "ansible",
"playbook_file": "./playbook.yml",
"extra_arguments": [
"--extra-vars",
"docker_pkg_name='docker-engine={{user `docker_version`}}-0~ubuntu-trusty'"
]
}
]
}

The contents of our playbook.yml is as follows:

---
- name: Install Docker on Ubuntu 14.04
hosts: all
# run as root
become: true
become_user: root

pre_tasks:
- name: format the extra drive
filesystem:
dev: /dev/xvdb
fstype: ext4
- name: mount the extra drive
mount:
name: /secondary
# ubuntu renames the block devices to xv* prefix
src: /dev/xvdb
fstype: ext4
state: mounted
roles:
- role: angstwad.docker_ubuntu
docker_opts: "--graph /secondary --storage-driver=aufs"

Prior to running the tool, we will need to grab the Docker installation role at the root directory containing ubuntu_1404_aufs3.json, and run ansible-galaxy install angstwad.docker_ubuntu -p to download a pre-configured Docker installation role. The popular angstwad.docker_ubuntu role exposes a lot of options for Docker installation on Ubuntu and follows the official Docker installation tutorial closely. Finally, we execute the script below and await our new base image. The end result will be your base Docker image going forward.

 $> packer build ubuntu_1404_aufs3.json
 ... output
 ... output
 ==> amazon-ebs: Creating the AMI: Docker 1.12.4 Ubuntu 14.04 AUFS3 1486965623
 amazon-ebs: AMI: ami-1234abcd
 ==> amazon-ebs: Waiting for AMI to become ready...

AWS Infrastructure Creation

To start creating infrastructure components, please checkout the following repository for a Rancher architecture template on AWS Networking Layer Next up, most AWS services require setting up a VPC to provision services without errors. To do this, we will create a separate VPC with public subnets. The following provides a straight forward way to setup a standard template. Check out the networking module here. In main.tf, our entry file for the infrastructure we reference our network configurations from ./database, followed by passing in parameters into our module:

module "networking" {
source = "./networking"

aws_region = "${var.aws_region}"
tag_name = "${var.tag_name}"
aws_vpc_cidr = "${var.aws_vpc_cidr}"
aws_public_subnet_cidrs = "${var.aws_public_subnet_cidrs}"
}

You can now run the creation of our simple network layer.

terraform plan -target="module.networking"
... output ...
Plan: 6 to add, 0 to change, 0 to destroy.

$> terraform apply -target="module.networking"
... output ...
module.networking.aws_subnet.rancher_ha_c: Creation complete
module.networking.aws_subnet.rancher_ha_b: Creation complete
module.networking.aws_subnet.rancher_ha_a: Creation complete
module.networking.aws_route.rancher_ha: Creation complete

Apply complete! Resources: 6 added, 0 changed, 0 destroyed. HA Rancher Server Next up, let’s setup our networking and use our AMI to setup HA mode on Rancher. To start, we automate the HA setup of Rancher. With the latest update to HA process in 1.2+, Rancher no longer requires a bootstrap node and interdependent steps to put up a HA cluster. The new steps are:

• Create an External Database (RDS in this post)
• Create a free SSL cert for the HA loadbalancer
• Use an external loadbalancer to route between the 3 nodes (ELB in this post)
• Launch HA nodes with an additional flag --advertise-address and Port Forwarding on 9345.

With the removal of the bootstrap node, the automation of HA Rancher setup becomes much easier. Lets begin by creating our external database. Create an External Database Continuing in main.tf, we then stand up our RDS database.

 module "database" {
 source = "./database"

vpc_id = "${module.networking.vpc_id}"
 database_subnet_ids = [
 "${module.networking.vpc_subnet_a}",
 "${module.networking.vpc_subnet_b}",
 "${module.networking.vpc_subnet_c}",
 ]
 database_port = "${var.database_port}"
 database_name = "${var.database_name}"
 database_username = "${var.database_username}"
 database_password = "${var.database_password}"
 database_instance_class = "${var.database_instance_class}"
 }

The database will then create security groups that consist of subnets defined in our networking layer. You can see the complete database terraform template on GitHub.

 $> terraform plan -target="module.database"
 ... output ...
 Plan: 3 to add, 0 to change, 0 to destroy.

$> terraform apply -target="module.database"
 ... output ...
 module.database.aws_db_instance.rancherdb: Still creating... (4m20s elapsed)
 module.database.aws_db_instance.rancherdb: Creation complete

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Creating a Free Cert for our ELB For this walkthrough, we use AWS Certificate Manager (ACM) to manage a SSL cert for our Rancher HA cert. You can look up how to request a free SSL certificate on the ACM docs. The process of requesting a cert from ACM contains manual steps to verify the domain name, so we don’t automate this section. Once provisioned, referencing the SSL certificate is as simple as adding the following data resource, you can view the file on GitHub.

 data "aws_acm_certificate" "rancher_ha_cert" {
 domain = "${var.fqdn}"
 statuses = ["ISSUED"]
 }

Creating the HA Server Group Next up we create an ELB setup with its accompanying security groups. Afterwards, we will add in three EC2 hosts for Rancher to reside in.

 module "rancher_server_ha" {
 source = "./rancher_server_ha"

vpc_id = "${module.networking.vpc_id}"
 tag_name = "${var.tag_name}"

# ssled domain without protocol e.g. moo.test.com
 acm_cert_domain = "${var.acm_cert_domain}"
 # domain with protocol e.g. https://moo.test.com
 fqdn = "${var.fqdn}"

# ami that you created with packer
 ami = {
 us-east-1 = "ami-f05d91e6"
 }

subnet_ids = [
 "${module.networking.vpc_subnet_a}",
 "${module.networking.vpc_subnet_b}",
 "${module.networking.vpc_subnet_c}",
 ]

# database variables to be passed into Rancher Server Nodes
 database_port = "${var.database_port}"
 database_name = "${var.database_name}"
 database_username = "${var.database_username}"
 database_password = "${var.database_password}"
 database_endpoint = "${module.database.endpoint}"
 }

The details of the HA Server template creates security groups, ELBs, and autoscaling groups. This may take a few moments to stand up, as we will need to wait for EC2 instances to start up.

 $> terraform plan -target="module.rancher_server_ha"
 ... output ...
 Plan: 11 to add, 0 to change, 0 to destroy.

$> terraform apply -target="module.rancher_server_ha"
 ... output ...

Apply complete! Resources: 11 added, 0 changed, 0 destroyed.

[Cloud Config on HA Instance] We provision our server node resources inside ./files/userdata.template file. It essentially fills in variables to create a cloud-init config for our instance. The cloud init docs writes a file called start-rancher.sh and then executes it on instance start. You can view the details of the file here. [Point DNS at the ELB] Now you can point your DNS server at our Rancher ELB that we created. Navigate to ELB console from there you should see the created ELB. You then grab the DNS name for the ELB and on your domain name provider, add a CNAME record to it. For example, in this post, I setup Rancher on rancher.domain.com and then access the admin panel on https://rancher.domain.com.

Rancher Node Setup

At this point, we have already setup the Rancher server and we can add custom hosts or use the Rancher-provided hosts drivers. If we want to try more automation, here is a potential way to automate autoscaled slave node clusters on AWS. From the Rancher UI, we follow the documentation for adding custom hosts. We will need to grab a few variables to pass into our cluster setup template. At the time of writing the custom host command is:

 sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:${rancher_agent_version} ${rancher_reg_url}

# Example at the time of writing.
 rancher_reg_url = https://rancher.domain.com/v1/scripts/AAAAABBBBB123123:150000000000:X9asiBalinlkjaius91238
 rancher_agent_version = v1.2.0

After pulling a those variables, we can then run the node creation step. Since this is a separate process than setting up HA, in the file we initially comment out this the creation of the Rancher nodes.

 $> terraform plan -target="module.rancher_nodes"
 ... output ...
 Plan: 3 to add, 0 to change, 0 to destroy.

$> terraform apply -target="module.rancher_nodes"
 ... output ...

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

After a few moments, you should see your Rancher host show up in your Rancher UI.

Summary

That was a lot steps, but with this template, we can now build each Terraform component separately and iterate on the infrastructure layers, kind of like how Docker images are built up. The nice thing about all these various components is replaceability. If you don’t like the choice of OS for the Docker Host, then you can change up the Packer configurations and update the AMI ID in Terraform. If you don’t like the networking layer, then take a peek at the Terraform script to update it. This setup is just a starter template to get Rancher up and your projects started. By no means is this the best way to standup Rancher, but the layout of Terraform should allow for continuous improvement as your project takes off. Additional Improvements

• The VPC shown here resides in the public subnet (for simplicity), but if you want to secure the network traffic between the database and servers, you’ll need to update the networking (this would require a rebuild).
• We might be able to look into passing our Rancher nodes into a separate project instead of commenting it out.
• Also, we should take a look at how to backup state on Terraform in case we lose the folder for state. So a bit more setup into S3 backup would help for those who plan to use this in production.
• EFS can also be a candidate into the script to add distributed files system support to our various nodes.
• Cross region RDS replication: terraform-community-modules/tf\_aws\_rds
• Use Terraform VPC module managed by the terraform community: terraform-community-modules/tf\_aws\_vpc

Collection of Reference Architectures

There are many reference architectures from various community members and Rancher contributors that are created by the community. They are further references after testing this template, and you can reference their structures to improve on the infrastructure.

Terraform

• github.com/mlaccetti/terraform-aws-rancher-ha
• https://github.com/cloudnautique/terraform-rancher

For advanced networking variants, there is also a Cloudformation reference here Nick Ma is an Infrastructure Engineer who blogs about Rancher and Open Source. You can visit Nick’s blog, CodeSheppard.com, to catch up on practical guides for keeping your services sane and reliable with open-source solutions.