关注微信公众号
第一手干货与资讯
加入官方微信群
获取免费技术支持
Recently, Rancher released a community catalog that will contain entries of Compose templates generated by the community. By default, the catalog in Rancher UI is populated from the Rancher catalog repository under the name “library catalog”. Now, you can also see the community catalog as well. This post will introduce how to build a secure Consul cluster as a Rancher Compose template that will be an addition to the newly released Rancher community catalog. Consul is a service discovery tool created by Hashicorp. It is a widely distributed and readily available tool. The features and capabilities of Consul include:
Security is considered the primary concern when it comes to provisioning a Consul cluster in production. Consul has a multi-faceted approach. Gossip messaging is secured using a symmetric 16-byte key and RPC communication between the nodes can be secured using TLS with optional authenticity checks for servers and clients.
The new templates have to follow a set of rules. Each template must be in a separate folder, preferably with a simple name as recommended by Rancher. Inside this folder, you should have a subfolder for each version you create using these templates, The first version should be 0 and each subsequent version will be an incremental value such as 0.1 and so on.: The versioning system enables you to upgrade your stack easily from a previous version of the template. Each version requires two yml files; docker-compose.yml and rancher-compose.yml. The docker-compose file is the file that defines services, networks, and volumes for the Compose template. The Docker Compose file should be launched using the “docker-compose up” command. For more information about using the Docker- Compose file, please refer to the official Docker documentation. The second file is the Rancher-Compose file that contains additional information about the services that will run on top of Rancher, such as the scale of service. This file also requires a .catalog section which defines relevant information about the catalog template, the .catalog content should look like this:
.catalog name: # Name of the versioned template of the Catalog Entry version: # Version of the versioned template of the Catalog Entry description: # Description of the versioned template of the Catalog Entry uuid: # Unique identifier to be used for upgrades. minimum_rancher_version: # The minimum version of Rancher that supports the template questions: #Used to request user input for configuration options
The questions item under the .catalog section will be used to change the configuration options for the template entry. For more information about the questions section, please refer to Rancher’s official documentation. We will see in the next section how to use the answers to these questions to change the values of the Docker containers in the catalog entry. Outside the version directory, you should have a picture that will be displayed in the Rancher UI catalog, and a config.yml file, which will contain information on how to display the catalog entry:
name: # Name of the Catalog Entry description: | # Description of the Catalog Entry version: # Version of the Catalog to be used category: # Category to be used for searching catalog entries maintainer: # The maintainer of the catalog entry license: # The license projectURL: # A URL related to the catalog entry
The consul template has the following docker-compose file:
consul-conf: image: husseingalal/consul-config labels: io.rancher.container.hostname_override: container_name volumes_from: - consul net: "container:consul" consul: image: husseingalal/consul labels: io.rancher.sidekicks: consul-conf volumes: - /opt/rancher/ssl - /opt/rancher/config - /var/consul
The template will consist of two docker images, the consul image is 40MB, which is based on gliderlabs/consul-agent image, and the second image is consul-config 10MB image. The consul image is in the label section:
io.rancher.sidekicks: consul-conf
The consul-conf file instructs Rancher to use consul-conf as a sidekick for the consul Docker image. The sidekick containers are secondary containers that are scheduled with the primary container. These secondary containers can share volumes and namespaces to build a single unit. It can be used for different purposes like defining volumes and then referencing these volumes in the primary container. For more information about Sidekicks, check out the official documentation of Rancher. The image uses confd to generate the configuration files from the templates. The Rancher team have developed a Rancher Metadata backend for Confd. The metadata backend will provide a lot of information about the services and the containers including the answers to the questions that will be defined in rancher compose file. Confd is a lightweight tool that watches a key/value store, like Rancher’s metadata backend. It is used to update source templates with the value it gets from the key/value (k/v) store. Confd supports two modes of running, “one time” which processes the template from the k/v store one time and exits, and the second mode is the “daemon mode” which will start in the background and will periodically check the backend for any change in the values to update the source templates. The Rancher-Compose file will define the questions that will be populated within the Docker containers in the service:
.catalog: name: "Consul" description: "Secure Consul cluster" version: "0.6-rancher1" uuid: consul-0 questions: - variable: ca_crt label: "CA certificate" type: "multiline" required: true - variable: consul1_key label: "First consul key" type: "multiline" required: true - variable: consul1_crt label: "First consul certificate" type: "multiline" required: true - variable: consul2_key label: "Second consul key" type: "multiline" required: true - variable: consul2_crt label: "Second consul certificate" type: "multiline" required: true - variable: consul3_key label: "Third consul key" type: "multiline" required: true - variable: consul3_crt label: "Third consul certificate" type: "multiline" required: true - variable: gossip_key label: "Communication gossip key" type: "multiline" required: true consul: scale: 3 metadata: ca.crt: | ${ca_crt} consul1.crt: | ${consul1_crt} consul1.key: | ${consul1_key} consul2.crt: | ${consul2_crt} consul2.key: | ${consul2_key} consul3.crt: | ${consul3_crt} consul3.key: | ${consul3_key} enc.key: "${gossip_key}"
The scale section has the value 3, which instructs Rancher to spin up only three containers for the consul service, and will add some values to the metadata backend of Rancher. For example, the ca.crt will be found under:
http://rancher-metadata/2015-07-25/service/consul/metadata/ca.crt
And the value \${ca_crt} is the answer to the question defined above:
questions: - variable: ca_crt label: "CA certificate" type: "multiline" required: true
The questions include:
These questions will make sure that the communication between the consul node is secured. Let’s take a closer look at how confd will use the metadata backend to generate the configuration file for the consul nodes.
The Docker file will be like the following:
FROM rancher/confd-base:0.11.0-dev-rancher ADD ./conf.d /etc/confd/conf. ADD ./templates /etc/confd/templates ENTRYPOINT ["/confd"] CMD ["--backend", "rancher", "--prefix", "/2015-07-25", "--log-level", "debug"]
The image is based on the Rancher/confd-base: 0.11.0-dev-rancher which is just a scratch image with the binary release of confd. The image will add the conf.d and templates directories which contain the resource templates configs for confd and the sources templates respectively. The source templates are written using Golang text templates, which is a very powerful templating language and supports multiple functions for many purposes. The source template for the consul configuration file will be like the following: server.json.tmpl
{ {{ if (eq (getv "/self/container/service_index") "1") }}"bootstrap": true,{{else}} "retry_join": [{{ $containerName := getv "/self/service/containers/0"}}"{{getv (printf "/containers/%s/primary_ip" $containerName)}}"], "bootstrap": false,{{end}} "server": true, "datacenter": "dc1", "advertise_addr": "{{getv "/self/container/primary_ip"}}", "bind_addr": "{{getv "/self/container/primary_ip"}}", "client_addr": "{{getv "/self/container/primary_ip"}}", "data_dir": "/var/consul", "encrypt": "{{getv "/services/consul/metadata/enc.key"}}", "ca_file": "/opt/rancher/ssl/ca.crt", "cert_file": "/opt/rancher/ssl/consul.crt", "key_file": "/opt/rancher/ssl/consul.key", "verify_incoming": true, "verify_outgoing": true, "log_level": "INFO" }
The template uses an “if” condition to determine whether this container is the first to run in the service or not, it checks for:
/self/container/service_index
This represents the index value of the container within Rancher’s metadata service. In this case, we want only the first container in the service to have the value true to bootstrap. Otherwise, the bootstrap value will equal false, and the retry_join configuration item will be added, which has the value of the primary IP of the master container. Also the bind_addr, advertise_addr, and the client_addr will have the value of the Rancher Managed Network IP which is represented by the metadata item:
/self/container/primary_ip
The bind address will be used for internal cluster communication. However, the client address will be used to bind the client interfaces such as HTTP, DNS, and RPC servers to certain addresses. The verify_incoming and verify_outgoing options are used to enable authenticity checks on both servers and clients, which will use the certificates we entered earlier to ensure that the client or server is part of the cluster. The consul.key, consul.crt, and ca.crt will be populated by confd using other templates: ca.crt.tmpl
{{getv "/ca.crt"}}
consul.crt.tmpl
{{ $containerID := (getv "/self/container/service_index") }}{{ getv (printf "/services/consul/metadata/consul%s.crt" $containerID) }}
consul.key.tmpl
{{ $containerID := getv "/self/container/service_index" }}{{ getv (printf "/services/consul/metadata/consul%s.key" $containerID) }}
The destination of these templates is defined by the configuration files for confd. The files are written in TOML:
[template] src = "server.json.tmpl" dest = "/opt/rancher/config/server.json" owner = "root" mode = "0644" keys = [ "/self/service/containers", "/containers", "self/container/name", "/self/container/primary_ip", "/services/consul/metadata/enc.key", "/self/container/service_index", ]
The file requires the src template path as well as the destination where this file will be placed, and you will have to define the keys used inside the templates.
After installing the Rancher platform and registering hosts to the management system, click on Catalog tab and then choose the consul template from there: If you scroll down you will see a bunch of questions that ask for the certificates and keys to secure the consul cluster: After adding these values, click deploy and wait for a minute until all containers are up and running: The six containers represent the three containers of the consul cluster and the three sidekicks (consul-conf) that will update the configuration for Consul server. You can make sure that the cluster is up and running by querying the DNS or HTTP interface of the consul cluster:
# curl http://10.42.142.182:8500/v1/status/peers ["10.42.231.143:8300","10.42.142.182:8300","10.42.205.12:8300"] # dig @10.42.231.143 -p 8600 consul.service.dc1.consul ; <<>> DiG 9.9.5-3ubuntu0.7-Ubuntu <<>> @10.42.231.143 -p 8600 consul.service.dc1.consul ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3550 ;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;consul.service.dc1.consul. IN A ;; ANSWER SECTION: consul.service.dc1.consul. 0 IN A 10.42.205.12 consul.service.dc1.consul. 0 IN A 10.42.142.182 consul.service.dc1.consul. 0 IN A 10.42.231.143 ;; Query time: 26 msec ;; SERVER: 10.42.231.143#8600(10.42.231.143) ;; WHEN: Fri Feb 19 13:50:22 EST 2016 ;; MSG SIZE rcvd: 166
As you can see by the instructions in this article, it’s not difficult to develop new catalog entries that meets the needs of your application. The Rancher community catalog provides the opportunity for open source enthusiasts to develop new catalog entries and share them with the open source community. Come on down to the Rancher/Docker environment and help build new stacks on top of Rancher.