Managing Docker Logging with Loggly

Once any application, dockerized or otherwise, reaches production, log aggregation becomes one of the biggest concerns. We will be looking at a number of solutions for gathering and parsing application logs from docker containers running on multiple hosts. This will include using a third-party service such as Loggly for getting setup quickly as well as bringing up an ELK stack (Elastic Search, Log Stash, Kibana) stack. We will look at using middleware such as FluentD to gather logs from Docker containers which can then be routed to one of the hundreds of consumers supported by fluentd. In this article we focus on using third party tools for Docker Logging, specifically using Loggly as an example. We will highlight how to get application logs to Loggly using both Docker and Rancher.

If you want to get setup aggregating logs quickly the best option is to use a hosted third-party solution. There are many such solutions for example Paper Trail, Splunk Cloudand Loggly. We will be using Loggly as an example, however all three platforms support similar ingestion interfaces. One option is to implement loggly integration directly into your application. For example if you are using Java you can use Logback. However, for a more general solution we can setup integration through the rsyslog daemon. This allows you to use the same setup regardless of application and language as syslog support is available in the vast majority of languages and platforms. Further, syslog allows you to configure local filtering and sampling to reduce the amount of logs you will send on to Loggly. This is important as Logging services tend to get very expensive for large volumes. Lastly, syslog integration can be used with many other logging solutions, so if you choose to switch from loggly to another option at a later point your application code does not have to change.

Before we get started with the actual integration lets sign up for a free Loggly trial account from the sign up page. Once you have logged into an account browse to Source Setup > Customer Token (https://YOUR_DOMAIN.loggly.com/tokens) to find your customer token. With this token we can send logs to Loggly for indexing and storage. In order to actually send logs we will be using the loggly-docker container. This container will setup and run a rsyslog daemon using your customer token and send them to Loggly. The container accepts incoming logs both from a unix domain socket as well as a UDP network port. If you are going to run the container on docker you can use the command shown below:

docker run --name loggly           \
    -e TOKEN=[YOUR_LOGGLY_TOKEN]   \
    -e TAG=[ANY_TAG]               \
    -v /tmp/:/var/run/loggly       \
    -p 524:524/udp                 \
    rancher/loggly-docker

From Rancher, use the Rancher compose and Docker compose templates shown below to bring up a stack with a logging service. This setup will launch a Loggly container on each Rancher compute agent connected to your environment.

# docker-compose.yaml
logging:
  environment:
    TOKEN: [YOUR_LOGGLY_TOKEN]
    TAG: [ANY_TAG]
  labels: io.rancher.scheduler.global: 'true'
  tty: true
  image: rancher/loggly-docker
  volumes: - /tmp:/var/run/loggly
  ports:
  - 514:514/udp
  stdin_open: true

# rancher-compose.yaml
logging:
  scale: 1

Once you container(s) are up and running your can test your logging setup by using the following command:

# Test Unix Socket
ls -l /tmp/socket/log
echo netcat:"Docker-Test My Log Message" | nc -u -w 1 -U /tmp/socket/log

In a few seconds your log messages should be visible in the loggly console. Note that the TAG you specified in the docker command or compose template should be parsed out as well as the application name.

Now that we have the unix socket and the network socket setup we can use it from other containers. To do so we can volume mount our unix socket into the container and setup the application running inside to use syslog based logging. To serve as an example for this setup we will use our golang based sample application go-auth. We will run the go-auth container, mount the syslog socket into it and configure the application to write to the socket. To run the go-auth container using Docker use the following command. This will run the usman/go-auth container which is defined here, and specify that it should run with syslog based logging, with debug level logs, on port 9000 with the database running at MYSQL_HOSTNAME.

Note that we mount the directory containing the socket on the host machine instead of the socket itself. Due to issues in the way Docker volumes are mapped you cannot mount the socket itself. Further we must mount the socket into a location like /tmp and then symlink it to the correct place, /dev/log, (The container already handles this) because mounting the directly to /dev will overwrite other systems files which are needed. Our earlier setup ensures that there is a loggly-docker container running on every compute node hence regardless of where our sample app launches containers the /tmp/socket directory will be available for volume mounting.

docker run                      \
  -v /tmp/socket/:/tmp/         \
  -p 9000:9000                  \
  usman/go-auth                 \
  -t syslog -l debug run -p 9000 --db-host MYSQL_HOSTNAME

The sample application requires a running instance of MySQL which can be setup inside a Docker container with the following command. If you do use the containerized MySQL you would need to link it to the go-auth container listed above.

docker run -d --name mysql -e MYSQL_ROOT_PASSWORD=SOME_PASSWORD \
    -e MYSQL_DATABASE=messenger                                 \
    -e MYSQL_USER=messenger                                     \
    -e MYSQL_PASSWORD=messenger                                 \
    -p 3306:3306 mysql

Once you have setup the go-auth container you can register a new user and generate an authentication token for the user with the following two commands.

# Add new User
curl -i -X PUT -d userid=USERNAME -d password=PASSWORD http://GO-AUTH-HOSTNAME:9000/user

# Generate Auth Token
curl 'http://GO-AUTH-HOSTNAME:9000/token?userid=USERNAME&password=PASSWORD'

You can also create an instance of the sample application and requisite MySQL container using the following docker compose template with a trivial rancher-compose.yaml to specify a scale of one for both mysql and go-auth services.

# docker-compose.yaml
GoAuth:
  command:
    - -t
    - syslog
    - -l
    - debug
    - run
    - -p
    - '9000'
    - --db-host
    - mysql
  image: usman/go-auth
  links
    - mysql:mysql
  volumes:
    - /tmp/socket:/tmp/
 stdin_open: true
 tty: true

mysql:
  restart: 'no'
  environment:
    MYSQL_ROOT_PASSWORD:rootpass
    MYSQL_DATABASE:messenger
    MYSQL_USER:messenger
    MYSQL_PASSWORD:messenger
  image: mysql
  stdin_open: true
  tty: true

# rancher-compose.yaml
GoAuth:
  scale: 1
mysql:
  scale: 1

Regardless of how you launch your linked container after a few seconds you should start seeing logs like the one shown below in the Loggly console. Note that the same socket can be bound to multiple containers. Furthermore since we used Rancher global scheduling to launch a Loggly container on each host we can rely on the /tmp/socket/log file being setup regardless of which host your go-auth container gets launched on. This setup will keep the configuration for syslog in one place (inside container image and launched using a single Rancher service) while distributing the load of logging across all hosts in the cluster.

Once you have your data in a log aggregation system we can do several interesting types of analysis. First we can see the distribution of events across our code base. For example the search below shows the distribution of error logs across various classes in our code base. This will help us target efforts to fix bugs as rare bugs can be prioritized lower. In most of my projects we send several properties with each log message including Service Name, Class Name, Method Name as well as some client side properties such as Platform (iOS, Android etc) so that we can target development team effort according to real need.

Another common use case for log parsing is the ability to isolate all logs for a given request. We tag all our log events with the Request ID regardless of which server or service is logging the message. This way when we come across a bug we can see the entire life-cycle of the problematic interaction across various services. This is even more important in dockerized applications which are likely to be micro services. To get an idea of what the user-flow was, getting logs from all the containers and services in a form that we can correlate is essential.

Lastly, another common question that comes up when we find a bug is \“Is this a new problem?\” Hence having a record going back in time is essential. With this kind of log aggregation we can see trends of how often specific events were happening. This information can be correlated with releases to find out which code change caused the issue. For example in the screen shot below, the error retrieving the webview page seems to have been around for at least since August 10th.

Although it is not the primary motivator we can even use such trends to setup automated alerts. For example the setup below will trigger an alert if logs tagged with the Level Error happen at a frequency of more than 10 in a given 5 minute period. This is a very coarse grained alert which could probably have been setup more easily in a traditional monitoring setup however more detailed search such as \“a specific error messages happening for the same user at a certain rate\” can be very difficult to emulate in a metrics system.

We have seen how easy it is to get Docker Logging and aggregation up and running with a third party solution such as Loggly. For most small scale projects and teams this is the ideal solution as it provides a full featured log aggregation and indexing system with little investment of time and team resources. For your effort you get much deeper understanding of how your system is performing under real load and an invaluable tool in debugging problems. Reading logs from individual servers separately would make it impossible to find or analyze issues in a distributed system especially if you use micro-services architecture.

However, as your setup grows Loggly and similar tools can get expensive. In the next article we will look at the use of in-house solutions using Elastic Search and Kiabana. This can be cheaper for large volumes of logs but requires expertise and resources to setup and maintain a the system. We will present a proof of concept setup however, setting up and maintaining a large scale distributed log aggregation system is a non-trivial exercise.

If you’d like to learn more about Docker Logging best practices with Rancher, please view a recordingof our recent online meetup discussing how to deploy an ELK stack on Rancher for monitoring containers. Also, please join us for our next meetup in October, where we will be demonstrating best practices for monitoring Docker containers.

Usman is a server and infrastructure engineer, with experience in building large scale distributed services on top of various cloud platforms. You can read more of his work at techtraits.com, or follow him on twitter @usman_ismailor on GitHub.