Building a Rancher Container Service on Mesos

*This post is now a bit out of date. Since posting this article we’ve added full support for Mesos environments directly into Rancher. You can read more about it at rancher.com/mesos. * Hi, I’m Sidhartha Mani, one of the engineers here at Rancher Labs. Over the last few months I’ve been working with Apache Mesos, an open source resource manager and scheduler, which can be used to deploy workloads on infrastructure. Mesos is very good at understanding resources and capacity management, but for more advanced workload or container management, many users choose a framework such as Marathon. Frameworks provide capabilities like load balancing, service discovery, rolling upgrades, application composability, continuous deployment and others. A number of Rancher’s community members who are also using Mesos have felt the need for a system that combines the fault tolerance, and scheduling capabilities of Mesos with the multi-tenant, private container service capabilities provided by Rancher. One community user, Marcel Neuhausler from AT&T Foundry, took the initiative to chart a broad design and envisioned a workflow for such an integration. He had a very interesting insight to combine Mesos’ ability to schedule VMs with Rancher’s ability to then manage those VMs and deploy containers on them. He wrote a Mesos framework for Rancher on Mesos, which proved to be a great starting point for writing the framework I’m going to talk about today. In this blog, I am going to describe the ideas and the software - Rancher Mesos framework, that resulted through the act of collaboration with Marcel. This framework can be used to setup large scale production jobs like Hadoop, Kafka, ElasticSearch etc. in docker containers, as well as any part of the DevOps pipeline, such as build, test, staging or production application environments. In the sections below, I’ll discuss the architecture of the framework, and show you how to set it up on your local environment.

[](https://gist.github.com/wlan0/b5379d9d727579496c5b#the-rancher-mesos-architecture)The Rancher Mesos Architecture

Here’s a diagram explaining the Rancher Mesos Framework As you can see, Rancher integrates with Mesos using standard Mesos interfaces - A Mesos framework. The various components are

1. Mesos Master: The Mesos Master is a cluster of machines that run the mesos-master process. It maintains, and monitors the Mesos slaves, and handles resource offering, task launching, task monitoring, fault tolerance and message passing etc.
2. Mesos Slave: These are the hosts on which jobs are to be scheduled. In this case, these will be used to launch VMs that register with Rancher. We use VMs instead of containers because we provision hosts using Mesos, unlike other frameworks that schedule jobs on it. These VMs can then be orchestrated using Rancher, and jobs can be scheduled on them using containers.
3. Rancher Server: This is a cluster of machines that run the rancher/server docker container. It maintains, and monitors the Rancher host, while providing a multi-tenant container service to users which includes a number of container management and infrastructure features, such as private deployment environments, container networking, Docker compose support, service discovery, rolling upgrades, registry management and more.
4. Rancher Hosts: These are hosts provisioned using Mesos’ resource offers. These hosts run docker and have therancher/agent container running, which is used for Rancher’s private networking, container scheduling and for various tasks involving the hosts.
5. Rancher-Mesos Scheduler (github): The scheduler is a two tiered application. It is a Rancher external event handler, as well as a Mesos scheduler. The event handler is used to listen on create host event from Rancher. The scheduler is used to listen for resource offers from Mesos. When the Rancher-Mesos scheduler receives a create host event, it adds that event to an event queue. Once Mesos provides a suitable slave to schedule tasks on, it de-queues events , and the Rancher Host is created on that Mesos slave, if it has sufficient capacity.
6. Rancher-Mesos Executor (github): This is the process that is invoked when an available slave is provided to Rancher for creating hosts. This process uses QEMU-KVM to create VMs with bridge networking. Docker is installed on these VMs and then rancher/agent is started to make it register with Rancher Server.
7. Rancher-Mesos Framework: The Rancher-Mesos Framework is used to refer to Rancher-Mesos Scheduler and Rancher-Mesos Executor collectively.

[](https://gist.github.com/wlan0/b5379d9d727579496c5b#the-rancher-mesos-workflow)The Rancher Mesos Workflow

The user’s point of view of working with the Rancher Mesos framework would be no different from using Rancher today.

1. The user would click on Add Host in the UI, which would provision a host in one of the available Mesos slaves. The slave on which it is provisioned is determined by the Mesos master.
2. The host, once provisioned will register itself with the rancher server. It will show up in the UI and the user can view stats, execute shell or start/stop containers like normal.

[](https://gist.github.com/wlan0/b5379d9d727579496c5b#looking-under-the-hood)Looking under the hood

This picture explains the sequence of events to provision a host using Rancher Mesos

1. When you click on Add Host in the UI, Rancher server creates a physicalhost.create event.
2. This event is received by all the external handlers that have subscribed to this event. In this case, the Rancher-Mesos scheduler subscribes to this event.
3. On Receiving the event, the scheduler saves the event in an event queue.
4. Then the scheduler waits for a resource offer of a free host from Mesos Master.
5. Once the scheduler receives the resource offer, it can retrieve the earliest event from the queue, and launch that task on the offered host.
6. The task starts Rancher Mesos Executor. The executor uses QEMU-KVM to start a new VM.
7. Then it install docker on the new VM.
8. The executor then instructs the new VM to registers itself as a host with rancher server.

[](https://gist.github.com/wlan0/b5379d9d727579496c5b#setting-up-and-running-rancher-mesos-framework)Setting up and running Rancher Mesos framework

In this section, I’ll show you how to setup this architecture on your laptop to try it out. We’ll use VMware fusion pro to virtualize the setup as it requires changing networking configuration, and its easier to work this way. Download the iso for Ubuntu Desktop 14.04.2. In VMware fusion, select Add > Install from disk or image. Make sure you enable nested virtualization, and have at least 2GB of Memory before booting. To enable nested virtualization

    Click on settings >
        Processors and Memory >
            Advanced Options >
                Enable Hypervisor Applications

Now boot it up.

1. The first step in setting up is network configuration. We need to setup bridge networking for eth0. Before continuing, ensure that bridge-utils is installed, usingsudo apt-get install bridge-utils. Setup your /etc/network/interfaces as follows :-

   auto lo
   iface lo inet loopback
   
   auto eth0
   iface eth0 inet manual
   
   auto br0
   iface br0 inet dhcp
       bridge_ports eth0
       bridge_stp off
       bridge_fd 0
       bridge_maxwait 0

   Then run ifup -a, which reads the config file and sets up the bridge interface. If you run ifconfig now, you’ll notice there is no IP address on eth0, and there is a br0interface with a configured IP address. From here onwards, when I refer to $IP, it is the IP address on br0 on this machine

2. The next step is installing the necessary packages. First, you’ll need gitsudo apt-get install gitTo install QEMU-KVM, use this command,

   sudo apt-get install -y qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
   echo 'allow br0' > /etc/qemu/bridge.conf

   Then, install the executor (You need golang, mercurial , and Godeps)

   go get -d github.com/wlan0/rancher-mesos-executor
   cd $GOPATH/src/github.com/wlan0/rancher-mesos-executor && ./scripts/build
   sudo cp build/rancher-mesos-executor /bin/

   Then replace executor with scheduler in the previous steps to install rancher-mesos-scheduler. Finally, install docker

   wget -qO- https://get.docker.com | sh

3. Start rancher-server

   sudo docker run -p 8080:8080 -d wlan0/rancher-server

   This will start rancher-server on port 8080

4. Install Mesos master and slave

   sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF
   echo "deb http://repos.mesosphere.io/ubuntu/ trusty main" > /etc/apt/sources.list.d/mesosphere.list
   sudo apt-get -y update
   sudo apt-get -y install mesos
   service zookeeper stop
   sudo apt-get -y remove --purge zookeeper
   echo manual > /etc/init/mesos-master.override

5. Start Mesos master and slave

   sudo nohup mesos-master --work_dir=$(pwd) --ip=$IP &
   sudo nohup mesos-slave --master=$IP:5050 --ip=$IP &

6. Start the rancher-mesos scheduler.

   CATTLE_URL=http://$IP:8080/v1 CATTLE_ACCESS_KEY=service CATTLE_SECRET_KEY=servicepass MESOS_MASTER=$IP:5050 IP_CIDR=$IP/24 rancher-mesos-scheduler

7. Now from a browser, go to $IP:8080 to see the Rancher UI. Now, I had to do a bit of a hack to get the UI to call to mesos, as Rancher uses Docker Machine for adding resources from clouds, and I haven’t had time to create a custom Mesos machine driver. So, to add a host, click on add host within any environment and select the \“Rackspace\” icon, use any dummy credentials, and hit create. You should see a host get added to the Infrastructure tab of Rancher. Wait for a few minutes for the host to connect to cattle. Once it does, you’ll be able to use this host to start containers. Note: I have short circuited the authentication part in the external handler(rancher-mesos-scheduler) to ignore the cloud type and always provision Mesos hosts, so this will work from any of the different cloud drivers or from the Rancher API. In the future I’ll add a proper Mesos driver for the create host function.

8. Note that everytime you provision a host, the console for the created VM will pop up on your screen. This can be disabled for production environments.

Hopefully this gives you an idea of how to deploy Rancher as a framework on Mesos. Thanks again to Marcel from AT&T Foundry for workign with our team on this, and all the other community members who have attempted or suggested this integration. With Rancher on Mesos, creating a multi-tenant private container service, on top of your Mesos cluster.

[](https://gist.github.com/wlan0/b5379d9d727579496c5b#next-steps)Next Steps

1. If you have any questions, please post them on our forums
2. If you like to reach out to me, or have any questions, email me at sid@rancher.com

Also, If you’d like help setting up your environment please join our Rancher beta program, request a demo, or register for our next online meetup.