Running Collectd as a System Service on RancherOS

One of the exciting things about RancherOS is the concept of running system services as containers. It offers the chance to clearly delineate between containers running an application, and containers running agents and operating system services. This has some interesting potential implications for managing operations, such as making patching and upgrading system services simpler, setting app and organizational policies for required services, and prioritizing which services have access to system resources.

In this post, I will show how to install a typical system monitoring agent, Collectd, as a system container on RancherOS. If you’re not familiar with it, RancherOS is a lightweight Linux distro designed from the ground up to run Docker containers.

Collectd is an open-source tool that gathers and stores information about the system environment. It is generally used to gather statistics about several system components, including disk usage, CPU load, and network interfaces. Collectd expands the area of monitoring by using a plugin system to gather information about several services and common software like Nginx, Apache, SMTP, and MySQL.

Collectd doesn’t generate graphs from the statistics it gathers, instead it writes the data to RRD files that can be used by many utilities to generate detailed graphs. However, Collectd does provide a small perl script that can use RRD files to generate simple graphs, I will be using this and then serving the graphs generated from this script on Nginx. For more information about Collectd visit the Collectd documentation.

Collectd Docker Image

First, we will create the Collectd docker image that will run as our system service. The directory structure of the Docker image will be as following:**** ****

collectd-nginx/
├── conf
│   ├── collectd.conf
│   ├── nginx-collectd.conf
│   └── nginx.conf
├── Dockerfile
└── run.sh

The Dockerfile will contain the instructions for creating a Docker image with Nginx and Collectd installed:

FROM ubuntu:14.04
MAINTAINER hussein.galal.ahmed.11@gmail.com
ENV DEBIAN_FRONTEND noninteractive

# Install nginx, collectd, and other dependencies
RUN apt-get -qq update
RUN apt-get install -yqq nginx wget collectd perl libconfig-general-perl librrds-perl libregexp-common-perl collectd-core libhtml-parser-perl librrd2-dev libsnmp-dev spawn-fcgi fcgiwrap

# Add nginx configuration
ADD conf/nginx-collectd.conf /etc/nginx/sites-available/
RUN rm /etc/nginx/sites-enabled/default
RUN ln -s /etc/nginx/sites-available/nginx-collectd.conf /etc/nginx/sites-enabled/nginx-collectd
ADD conf/nginx.conf /etc/nginx/nginx.conf

# Add the run script
ADD run.sh /tmp/run.sh

# Install Graphs Scripts
RUN cd /tmp/ && wget http://collectd.org/files/collectd-5.4.2.tar.gz
RUN cd /tmp/ && tar -xvf collectd-5.4.2.tar.gz
RUN mkdir -p /var/www/
RUN cp -r /tmp/collectd-5.4.2/contrib/collection3 /var/www/graphs
RUN chown -R www-data:www-data /var/www/graphs
WORKDIR /tmp/collectd-5.4.2

EXPOSE 80
ENTRYPOINT ["/bin/bash","/tmp/run.sh"]

The Dockerfile consists of 3 main parts:

1. Installing Nginx, Collectd, RRDtool, and other important packages like perl and fcgiwrap which will be used to execute the perl scripts.
2. Adding Nginx configuration files which are the main nginx.conf file and the virtualhost configuration file which will proxy the request to fcgiwrap to execute the cgi scripts.
3. Install the graphing scripts, and setting up the document root for nginx. Finally the Dockerfile will run the run.sh script to start the services:

run.sh

service collectd start
service fcgiwrap start
sleep 5
/usr/sbin/nginx

The collectd.conf will contain the configuration of the plugins that will be used by collectd daemon:

LoadPlugin cpu
LoadPlugin disk
LoadPlugin rrdtool
LoadPlugin memory
LoadPlugin load
LoadPlugin processes
LoadPlugin swap
LoadPlugin users
LoadPlugin interface

Visit the Github repo, to see the all the files used to build the Docker image. To run this image on RancherOS as a system Docker container execute the following command:

$ sudo system-docker run -d --net=host --name collectd --privileged husseingalal/collectd
fbdaa7b5d62edae37d6226e274718f29a801531b006d834e8907004544502338

This will install collectd on the system docker daemon that is running as PID1 in RancherOS. You can see the statistics collectd is gathering by accessing http://[ip-address-of-the-server]. The tool will provide the data for each enabled plugin gathered by the collectd daemon. The following are examples of common metrics gathered by Collectd: **System Load ** *Memory Utilization* **Disk Operations

   **

CPU Usage

Make the system container start at boot

Finally, for most system services you’ll want to have the service start at boot time. The final step in this walk through is to add the docker start line to /opt/rancher/bin/start.sh to make the container starts after each reboot to the system:

$ sudo mkdir -p /opt/rancher/bin
$ sudo echo “sudo system-docker start collectd” >> /opt/rancher/bin/start.sh
$ sudo chmod 755 /opt/rancher/bin/start.sh

#

Conclusion

RancherOS gives admins the option of separating system containers, from user containers. Running agents like collectd as system containers ensures that user activities don’t impact the performance of critical monitoring or access services. Using collectd gives us a good example of the three step process for creating a system container:

1. Build the Dockerfile for your system service and post it to Dockerhub.
2. Execute the \$ sudo system-docker run... command to launch the container as a system container.
3. Finally, if you want RancherOS to auto-start the container, add it to the start line.

You can download RancherOS and find more information our GitHub site, including a detailed getting started guide. If you would like to learn more about RancherOS and system services, please join our next online meetup to meet the team and learn about the latest with RancherOS. Hussein Galal is a Linux System Administrator, with experience in Linux, Unix, Networking, and open source technologies like Nginx, Apache, PHP-FPM, Passenger, MySQL, LXC, and Docker. You can follow Hussein on Twitter @galal_hussein.