Deploying Service Stacks from a Docker Registry and Rancher

Note: you can read the Part 1 and Part 2 of this series, which describes how to deploy service stacks from a private docker registry with Rancher. This is my third and final blog post, and follows part 2, where I stepped through the creation of a private, password-protected Docker registry. and integrated this private registry with Rancher. In this post, we will be putting this registry to work (although for speed, I will use public images). We will go through how to make stacks that reference public containers and then use them in Rancher to deploy your product. So let’s get started! First, we should understand the anatomy of a Rancher stack. It isn’t that difficult if you are familiar with Docker Compose YAML files. A Rancher stack needs a docker-compose.yml file, which is a standard Docker Compose formatted file, but you can add in Rancher-specific items, like targeting servers via labels. The rancher-compose.yml file is specific to Rancher. If it doesn’t exist, then Rancher will assume that each container that is specified in the docker-compose.yml file will have a scaling factor of 1. It is good practice to detail a rancher-compose.yml file. So we will attempt to do the following, and make extensive use of Rancher labels. (1) Create a stack that deploys out a simple haproxy + 2 nginx servers (2) Create an ELK stack to collect logs (3) Deploy a Logspout container to collate and send all Docker logs to our ELK stack. Here are the characteristics of each of our containers, and the Rancher-specific labels that help us achieve our goals:

Containers **Host Label ** Rancher Labels ElasticSearch + Logstash + Kibana Deploy only onto a specific host with label \“type=elk\” io.rancher.container.pull_image: always io.rancher.scheduler.affinity:host_label: type=elk Logspout Deploy onto all hosts unless labeled “type=elk” io.rancher.container.pull_image: always io.rancher.scheduler.global: true io.rancher.scheduler.affinity:host_label_ne: type=elk HAProxy + 2 nginx containers Deploy onto any host NOT labeled “type=elk” or “type=web1” or “type=web2” io.rancher.container.pull_image: always io.rancher.scheduler.affinity:host_label_ne: type=elk,type=web1,type=web2 nginx 1 Deploy onto the host labeled “type=web1” io.rancher.container.pull_image: always io.rancher.scheduler.affinity:host_label: type=web1 nginx 2 Deploy onto the host labeled “type=web2\” io.rancher.container.pull_image: always io.rancher.scheduler.affinity:host_label: type=web2

We will need 4 hosts in total for this and will deploy 2 stacks.

Stack 1: ELK + Logspout

ELK will be running without persistence. For me, that isn’t important as any logs that are older than a day are not very useful; if needed I can get the logs direct via the ‘docker logs’ command. The Logspout container will be required for every host, so we will use the Rancher label ‘io.rancher.scheduler.global: true’ to perform this. The ‘global: true’ should be pretty straightforward - it instructs Rancher to deploy this container to every available host in the environment. Below is the logspout definition. Alter the logspout command to the IP of your ELK host. Also provide the labels in the below screen to each of the hosts. Stack name: ELK Stack Description: My elk stack that will collect logs from logspout Docker-Compose.yml

elasticsearch:
  image: elasticsearch
  ports:
    - '9200:9200'
  labels:
    io.rancher.container.pull_image: always
    io.rancher.scheduler.affinity:host_label: type=elk
  container_name: elasticsearch
logstash:
  image: logstash
  ports
    - 25826:25826
    - 25826:25826/udp
  command: logstash agent --debug -e 'input {syslog {type => syslog port => 25826 } gelf {} } filter {if "docker/" in [program] {mutate {add_field => {"container_id" => "%{program}"} } mutate {gsub => ["container_id", "docker/", ""] } mutate {update => ["program", "docker"] } } } output { elasticsearch { hosts => ["elasticsearch"] } stdout {} }'
  links:
    - elasticsearch:elasticsearch
  labels:
    io.rancher.container.pull_image: always
    io.rancher.scheduler.affinity:host_label: type=elk
  container_name: logstash
kibana:
   image: kibana
   ports:
     - 5601:5601
   environment:
     - ELASTICSEARCH_URL=http://elasticsearch:9200
   links:
     - elasticsearch:elasticsearch
   labels:
     io.rancher.container.pull_image: always
     io.rancher.scheduler.affinity:host_label: type=elk
   container_name: kibana
logspout:
  labels:
    io.rancher.container.pull_image: always
    io.rancher.scheduler.global: true
    io.rancher.scheduler.affinity:host_label_ne: type=elk
  image: gliderlabs/logspout
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock
  container_name: logspout
  command: "syslog://111.222.333.444:25826"

Now click ‘Create’ and after a few seconds you will see the containers being created. After a few minutes, all containers will have been downloaded to the correct hosts as defined in the docker-compose YAML. At the end of this process, we should see that all hosts have been activated: Now we can verify that the containers are distributed to the correct hosts and that the logspout contianer is on all hosts apart from the elk host. So everything looks good. Let’s visit our kibana frontend to ELK @ 144.172.71.84:5601 All looks good. Now let’s get our next stack set up:

Stack 2: HAproxy + 2 Nginx containers

Stack name: Web Stack Description: HAProxy and 2 nginx containers, all logging to Elasticsearch Docker-Compose.yml

web1:
 image: tutum/hello-world
 container_name: web1
 labels:
 io.rancher.container.pull_image: always
 io.rancher.scheduler.affinity:host_label: type=web1
web2:
 image: tutum/hello-world
 container_name: web2
 labels:
 io.rancher.container.pull_image: always
 io.rancher.scheduler.affinity:host_label: type=web2
ha:
 image: tutum/haproxy
 ports:
 - 80:80
 - 443:443
 container_name: ha
 labels:
 io.rancher.container.pull_image: always
 io.rancher.scheduler.affinity:host_label_ne: type=elk,type=web1,type=web2
 links:
 - web1:web1
 - web2:web2

Now hit ‘Create’, and after a few minutes you should see the hello world nginx containers behind an HAProxy. If we look at our stacks page, we will see both stacks with green lights: And if we hit the ha IP on port 80 or 443, we will see the hello world screen. We can then hit refresh a few times and the second hostname will appear. You can then validate the hostnames by opening up the container name and executing the shell. We should do a final check of our hosts to see if we have distributed all of our containers as intended. Has our Elasticsearch instance received any logs from our Logspout containers? (You might have to create an index in Kibana first) Yay! Looks like we have been successful. To recap, we have deployed 2 stacks, and 9 containers across 4 hosts in a configuration that suits our requirements. The result is a service that ships all the logs of any new container automatically back to the ELK stack. You should now have enough know-how in Rancher to be able to deploy your own service stacks from your private registry. Good luck!