Riak Cluster Deployment Using Rancher And RancherOS on AWS

Recently I have been playing around with Riak and I wanted to get it running with Docker, using RancherOS and Rancher. If you’re not familiar with Riak, it is a distributed key/value store which is designed for high availability, fault tolerance, simplicity, and near-linear scalability. Riak is written in Erlang programming language and it runs on an Erlang virtual machine. Riak provides availability through replication and faster operations and more capacity through partitions, using the ring design to its cluster, hashed keys are partitioned by default to 64 partitions (or vnodes), each vnode will be assigned to one physical node as following: From Relational to Riak Whitepaper For example, if the cluster consists of 4 nodes: Node1, Node2, Node3, and Node4, we will count around the nodes assigning each vnode to a physical node until the all vnodes are accounted for, so in the previous figure, Riak used 32 partition with 4 node cluster so we get:

Node0 : [1, 5, 9, 13, 17, 21, 25, 29]
Node1 : [2, 6, 10, 14, 18, 22, 26, 30]
Node3 : [3, 7, 11, 15, 19, 23, 27, 31]
Node4 : [4, 8, 12, 16, 20, 24, 28, 32]

So how about replication? Every time a write process happens Raik will replicate the value to the next N vnodes, where N is the value of the n_val setting in Riak cluster. By default, N is 3. To explain this, assume we will use the default n_val value and we will use the previous cluster setup with 4 nodes and 32 partitions, now lets assume we will write a key/value to partition (vnode) 2 which is assigned to the second node then the value will be replicated to vnode 3 and vnode 4 which are assigned to the 3rd and 4th nodes respectively. For more information about Riak cluster, visit the official riak documentation. In this post, I am going to deploy Riak cluster using Docker on RancherOS, the setup will include:

• Five Docker containers as Riak nodes.
• Each Container will be on separate EC2 Instance.
• RancherOS will be installed on each EC2 instance.
• The whole setup will be managed using Rancher platform.

##

The Riak Docker Image

Before launching your EC2 instances and the Rancher platform, you should create the Riak Docker image that will run each instance. I used the implementation of Riak Docker image of hectcastro, although I added and removed some parts to become suitable to run on RancherOS. First the Dockerfile:

FROM phusion/baseimage:latest
MAINTAINER Hussein Galal hussein.galal.ahmed.11@gmail.com

RUN sed -i.bak 's/main$/main universe/' /etc/apt/sources.list
RUN apt-get update -qq && apt-get install -y software-properties-common && \
    apt-add-repository ppa:webupd8team/java -y && apt-get update -qq && \
    echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && \
    apt-get install -y oracle-java7-installer

# Install Riak
RUN curl https://packagecloud.io/install/repositories/basho/riak/script.deb | bash
RUN apt-get install -y riak

# Setup the Riak service
RUN mkdir -p /etc/service/riak
ADD scripts/riak.sh /etc/service/riak/run

RUN sed -i.bak 's/listener.http.internal = 127.0.0.1/listener.http.internal = 0.0.0.0/' /etc/riak/riak.conf && sed -i.bak 's/listener.protobuf.internal = 127.0.0.1/listener.protobuf.internal = 0.0.0.0/' /etc/riak/riak.conf && \
    echo "anti_entropy.concurrency_limit = 1" >> /etc/riak/riak.conf && \
    echo "javascript.map_pool_size = 0" >> /etc/riak/riak.conf && \
    echo "javascript.reduce_pool_size = 0" >> /etc/riak/riak.conf && \
    echo "javascript.hook_pool_size = 0" >> /etc/riak/riak.conf

# Add Automatic cluster support
ADD scripts/run.sh /etc/my_init.d/99_automatic_cluster.sh
RUN chmod u+x /etc/my_init.d/99_automatic_cluster.sh
RUN chmod u+x /etc/service/riak/run

# Enable insecure SSH key
RUN /usr/sbin/enable_insecure_key.sh

EXPOSE 22 8098 8087
CMD ["/sbin/my_init"]

A couple of notes on the previous Dockerfile. The phusion/baseimage is used as the Docker base image, 2 important scripts were added to the image (riak.sh, automatic_cluster.sh) which I will explain in a second, the ports 8098 and 8087 are used for HTTP and Protocol Buffers and finally ssh support through insecure key was added. The purpose of the riak.sh script is to start the Riak service and ensure that the node name is set correctly, while the automatic_cluster.sh script is used to join the node to the cluster only if the RIAK_JOINING_IP is set during the starting of the contianer. riak.sh

#! /bin/sh

# Ensure correct ownership and permissions on volumes
chown riak:riak /var/lib/riak /var/log/riak
chmod 755 /var/lib/riak /var/log/riak

# Open file descriptor limit
ulimit -n 4096
IP_ADDRESS=$(ip -o -4 addr list eth0 | awk '{print $4}' | cut -d/ -f1 | sed -n 2p)

# Ensure the Erlang node name is set correctly
sed -i.bak "s/riak@127.0.0.1/riak@${IP_ADDRESS}/" /etc/riak/riak.conf
rm  -rf /var/lib/riak/ring/*

# Start Riak
exec /sbin/setuser riak "$(ls -d /usr/lib/riak/erts*)/bin/run_erl" "/tmp/riak" \
"/var/log/riak" "exec /usr/sbin/riak console"

automatic_cluster.sh

#!/bin/sh
sleep 10
if env | grep -q "RIAK_JOINING_IP"; then
  # Join node to the cluster
  (sleep 5;riak-admin cluster join "riak@${RIAK_JOINING_IP}"  && echo -e "Node Joined The Cluster") &

  # Are we the last node to join?
  (sleep 8; if riak-admin member-status | egrep "joining|valid" | wc -l | grep -q "${RIAK_CLUSTER_SIZE}"; then
    riak-admin cluster plan  && riak-admin cluster commit && echo -e "\nCommiting The Changes..."
  fi) &
fi

Also note that RIAK_CLUSTER_SIZE is used to specify the size of the cluster used in this setup. We don’t need more than that to start the cluster, now build the image and push it to Docker Hub to be used later.

# docker build -t husseingalal/riak2 .
# docker push husseingalal/riak2

Launch Rancher Platform

The Rancher Management platform will be used manage the Docker containers on RancherOS instances. First you need to run Rancher platform on a machine using the following command:

# docker run -d -p 8080:8080 rancher/server

Create RancherOS EC2 Instances

RancherOS is available as an Amazon Web Services AMI, and can be easily run on EC2, the next step is to create 5 EC2 instances to setup the cluster: You will get something like that after creating five instances with Amazon AWS: After creating the five instances, its time to register each instance with Rancher by running the following command on each server:

[rancher@rancher ~]$ sudo docker run --rm -it --privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent http://<ip-address>:8080/v1/scripts/4E1D4A26B07A1539CD33:1426626000000:jZskPi71YEPSJo1uMISMEOpbUo

After running the previous command on each server you will see that the servers have been registered with Rancher:

Running The Riak cluster

The RIAK_CLUSTER_SIZE will provide the number of instances needed to be added to the cluster before committing the changes, its recommended to add 5 Riak nodes to the cluster of a production environment, although you can set the RIAK_CLUSTER_SIZE to more or fewer as needed. **** **** To create a Docker container using the Rancher platform, on any instance click on “Add Container”: On the first node you just need to specify the name of the container and select the Riak image, but for other Riak nodes you need to specify two more environment variables which will help the node to connect to the cluster the RIAK_JOINING_IP which tells the Riak node to connect to a node in the cluster and RIAK_CLUSTER_SIZE which used to specify the number of nodes joining the cluster:

Testing The Riak Cluster

From Rancher we can view the logs of the running containers, similar to using docker logs -f container-name. This allows us to see the logs of the Riak containers and ensure that everything is running as planned: At the last node you will see something different. Since the number of the node that joined the cluster matches the value of the environment variable RIAK_CLUSTER_SIZE, so the changes will be committed and the cluster will be up and running: To see that the nodes are connected to the cluster, you can write the following command inside the shell of any of the Riak containers:

# riak-admin member-status

And you will get the following output: This indicates that each node is a valid member of the cluster and acquire a roughly equal percentage of the ring. Now to test the cluster from outside the environment, you should map the ports of the Docker containers to the host’s ports, this can be achieved dynamically using Rancher platform: I already created and activated a bucket-type called “cluster,” which I used to test via the Riak HTTP API. You can see from below the environment is up and running now.

$ export RIAK=http://52.0.119.255:8098
$ curl -XPUT "$RIAK/types/cluster/buckets/rancher/keys/hello" \
-H "Content-Type:text/plain" \
-d "World.. Riak"


$ curl -i "$RIAK/types/cluster/buckets/rancher/keys/hello"
HTTP/1.1 200 OK
X-Riak-Vclock: a85hYGBgzGDKBVIcqZfePk3k6vPOYEpkzGNlYAroOseXBQA=
Vary: Accept-Encoding
Server: MochiWeb/1.1 WebMachine/1.10.5 (jokes are better explained)
url: </buckets/rancher>; rel="up"
Last-Modified: Fri, 27 Mar 2015 22:04:50 GMT
ETag: "4flAtEZ59hdYsKhSGVhKpZ"
Date: Fri, 27 Mar 2015 22:11:23 GMT
Content-Type: text/plain
Content-Length: 5

World.. Riak

Conclusion

Riak cluster provides a distributed, high available, and simple key-value store. Building the Riak cluster using RancherOS and Rancher platform provide docker management and networking capabilities, making installation quick and making it simple to upgrade and scale the environment in the future. You can download Riak here. To download Rancher or RancherOS please visit our GitHub site. You can find a detailed getting started guide for RancherOS on GitHub as well. If you would like to learn more, please join our next online meetup to meet the team and learn about the latest with Rancher and RancherOS. Hussein Galal is a Linux System Administrator, with experience in Linux, Unix, Networking, and open source technologies like Nginx, Apache, PHP-FPM, Passenger, MySQL, LXC, and Docker. You can follow Hussein on Twitter @galal_hussein.