Building a Super-Fast Docker CI/CD Pipeline with Rancher and DroneCI

At Higher Education, we’ve tested and used quite a few CI/CD tools for our Docker CI pipeline. Using Rancher and Drone CI has proven to be the simplest, fastest, and most enjoyable experience we’ve found to date. From the moment code is pushed/merged to a deployment branch, code is tested, built, and deployed to production in about half the time of cloud-hosted solutions - as little as three to five minutes (Some apps take longer due to a larger build/test process). Drone builds are also extremely easy for our developers to configure and maintain, and getting it setup on Rancher is just like everything else on Rancher - very simple.

Our Top Requirements for a CI/CD Pipeline

The CI/CD pipeline is really the core of the DevOps experience and is certainly the most impactful for our developers. From a developer perspective, the two things that matter most for a CI/CD pipeline are speed and simplicity. Speed is #1 on the list, because nothing’s worse than pushing out one line of code and waiting for 20 minutes for it to get to production. Or even worse…when there is a production issue, a developer pushes out a hot fix only to have company dollars continue to grow wings and fly away as your deployment pipeline churns. Simplicity is #2, because in an ideal world, developers can build and maintain their own application deployment configurations. This makes life easier for everyone in the long run. You certainly don’t want developers knocking on your (Slack) door every time their build fails for some reason.

Docker CI/CD Pipeline Speed Pain points

While immutable containers are far superior to maintaining stateful servers, they do have a few drawbacks – the biggest one being deployment speed: It’s slower to build and deploy a container image than to simply push code to an existing server. Here are all the places that a Docker deployment pipeline can spend time: 1) [CI: pull base image for application from Docker registry] 2) [CI: build test image (with test dependencies) and run tests] 3) [CI: build production image (no test dependencies)] 4) [CI: push application image to Docker Registry] 5) [Infrastructure: pull application image from Docker Registry] 6) [Stop old containers, start new ones] Depending on the size of your application and how long it takes to build, latency with the Docker registry (steps 1, 4, 5) is probably where most of your time will be spent during a Docker build. Application build time (steps 2, 3) might be a fixed variable, but it also might be drastically affected by the memory or CPU cores available to the build process. If you’re using a cloud-hosted CI solution, then you don’t have control over where the CI servers run (registry latency might be really slow) and you might not have control over the type of servers/instances running (application build might be slow). There will also be a lot of repeated work for every build such as downloading base images for every build.

Enter Drone CI

Drone runs on your Rancher infrastructure much like a tool like Jenkins would, but, unlike Jenkins, Drone is Docker-native - every part of your build process is a container. Running on your infrastructure speeds up the build process, since base images can be shared across builds or even projects. You can also avoid a LOT of latency if you push to a Docker registry that is on your own infrastructure such as ECR for AWS. Drone being Docker-native removes a lot of configuration friction as well. Anyone who’s had to configure Jenkins knows that this is a big plus. A standard Drone deployment does something like this:

• Run a container to notify Slack that a build has started
• Configure any base image for your “test” container, code gets injected and tests run in the container
• Run a container that builds and pushes your production image (to Docker Hub, AWS ECR, etc)
• Run a container that tells Rancher to upgrade a service
• Run a container to notify Slack that a build has completed/failed

A .drone.yml file looks strikingly similar to a docker-compose.yml file – just a list of containers. Since each step has a container dedicated to that task, configuration of that step is usually very simple.

Getting Drone Up and Running

The to do list here is simple:

• Register a new GitHub OAuth app
• Create a Drone environment in Rancher
• Add a “Drone Server” host and one or more “Drone Worker” hosts
  • Add a drone=server tag to the Drone Server host
• Run the Drone stack

The instance sizes are up to you - at Higher Education we prefer fewer, more powerful workers, since that results in faster builds. (We’ve found that one powerful worker tends to handle builds just fine for teams of seven) Once your drone servers are up, you can run this stack:

version: '2'
services:
  drone-server:
    image: drone/drone:0.5
    environment:
      DRONE_GITHUB: 'true'
      DRONE_GITHUB_CLIENT: <github client>
      DRONE_GITHUB_SECRET: <github secret>
      DRONE_OPEN: 'true'
      DRONE_ORGS: myGithubOrg
      DRONE_SECRET: <make up a secret!>
      DRONE_GITHUB_PRIVATE_MODE: 'true'
      DRONE_ADMIN: someuser,someotheruser,
      DRONE_DATABASE_DRIVER: mysql
      DRONE_DATABASE_DATASOURCE: user:password@tcp(databaseurl:3306)/drone?parseTime=true
    volumes:
    - /drone:/var/lib/drone/
    ports:
    - 80:8000/tcp
    labels:
      io.rancher.scheduler.affinity:host_label: drone=server
  drone-agent:
    image: drone/drone:0.5
    environment:
      DRONE_SECRET: <make up a secret!>
      DRONE_SERVER: ws://drone-server:8000/ws/broker
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    command:
    - agent
    labels:
      io.rancher.scheduler.affinity:host_label_ne: drone=server
      io.rancher.scheduler.global: 'true'

This will run one Drone Server on your drone=server host, and one drone agent on every other host in your environment. Backing Drone with MySQL via the DATABASE_DRIVER and DATASOURCE values are optional, but highly recommended. We use a small RDS instance. Once the stack is up and running, you can login to your Drone Server IP address and turn on a repo for builds (from the Account menu). You’ll notice that there’s really no configuration for each repo from the Drone UI. It all happens via a .drone.yml file checked into each repository.

Adding a Build Configuration

To build and test a node.js project, add a .drone.yml file to your repo that looks like this:

pipeline:
  build:
    image: node:6.10.0
    commands:
      - yarn install
      - yarn test

It’s simple and to-the-point, your build step simply sets the container image that the repository code gets put in and specifies the commands to run in that container. Anything else will be managed with Drone plugins, which are just containers designed for one task. Since plugins live in Docker Hub, you don’t install them, just add them to your .drone.yml file A more full-featured build like I mentioned above uses Slack, ECR, and Rancher plugins to create this .drone.yml:

pipeline:
  slack:
    image: plugins/slack
    webhook: <your slack webhook url>
    channel: deployments
    username: drone
    template: "<{{build.link}}|Deployment #{{build.number}} started> on <http://github.com/{{repo.owner}}/{{repo.name}}/tree/{{build.branch}}|{{repo.name}}:{{build.branch}}> by {{build.author}}"
    when:
      branch: [ master, staging ]
  build:
    image: <your base image, say node:6.10.0>
    commands:
      - yarn install
      - yarn test
    environment:
      - SOME_ENV_VAR=some-value
  ecr:
    image: plugins/ecr
    access_key: ${AWS_ACCESS_KEY_ID}
    secret_key: ${AWS_SECRET_ACCESS_KEY}
    repo: <your repo name>
    dockerfile: Dockerfile
    storage_path: /drone/docker
  rancher:
    image: peloton/drone-rancher
    url: <your rancher url>
    access_key: ${RANCHER_ACCESS_KEY}
    secret_key: ${RANCHER_SECRET_KEY}
    service: core/platform
    docker_image: <image to pull>
    confirm: true
    timeout: 240
  slack:
    image: plugins/slack
    webhook: <your slack webhook>
    channel: deployments
    username: drone
    when:
      branch: [ master, staging ]
      status: [ success, failure ]

While this may be 40 lines, it’s extremely readable and 80% of this is copy and paste from the Drone plugin docs. (Try doing all of these things in a cloud hosted CI platform and you’ll likely have a day’s worth of docs-reading ahead of you.) Notice how each plugin really doesn’t need much configuration. If you want to use Docker Hub instead of ECR, use the Docker plugin instead. That about it! In a few minutes, you can have a fully-functioning CD pipeline up and running. It’s also a good idea to use the Rancher Janitor catalog stack to keep your workers’ disk space from filling up, just know that the less-often you clean up, the faster your builds will be, as more layers will be cached. Will Stern is a Software Architect for HigherEducation and also provides Docker training through LearnCode.academy and O’Reilly Video Training.