Running Cassandra on Rancher

Apache Cassandra is a popular database technology which is gaining popularity these days. It provides adjustable consistency guarantees, is horizontally scalable, is built to be fault-tolerant and provides very low latency, (sub-millisecond) writes. This is why Cassandra is used heavily by large companies such as Facebook and Twitter. Furthermore, Cassandra uses application layer replication for its data which makes it ideal for a containerized environment. However, Cassandra, like most databases, assumes that database nodes are fairly static. This assumption is at odds with a containerized infrastructure, which caters to more dynamic environments. In this post, I will cover some of the challenges this poses for managing Cassandra and other database workloads in containerized environments.

Simple Cassandra Cluster Setup

A basic Cassandra deployment on Rancher is fairly straight forward thanks to the work done by Patric Boos to create the Rancher-Cassandra container image in the community catalog. The docker-compose.yaml file to create a service stack with the Cassandra service is shown below. Note that we are creating a volume at /var/lib/cassandra. This is because the union file system containers are very slow and not suitable for high IO use cases. By using a volume, we defer to the host file system, which should be much faster.

Cassandra:
  environment:
    RANCHER_ENABLE: 'true'
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: pboos/rancher-cassandra:3.1
  stdin_open: true
  volumes:
    - /var/lib/cassandra

In addition, we set the RANCHER_ENABLE environment variable to true. If this variable is set, the container will use the Rancher Meta Data Service to get the IPs of all containers in the service (other than the current container) and set them up as seeds. Seeds are very important for Cassandra and this setup has a few very important impacts. First, seeds help bootstrap the cluster because all new nodes must report to the seeds so that they can give these new nodes a list of all the current servers and also tell the older connected servers about the new servers. If any server in the cluster did not know about any other, you would not be able to provide consistency guarantees. This is why Cassandra will refuse to start up if it cannot talk to all the seed servers. If you had multiple nodes launching at the same time, they would see each other as seeds. Since the nodes are not ready to accept connections, they would both fail to start, stating that they cannot connect to the seed. You can add more containers at a later time (once the first container is ready to serve traffic) but, you must do so one at a time and wait for that container to be ready before launching others.

In our fork of the container image, (usman/docker-rancher-cassandra:3.1) I slightly change the behavior by marking all nodes in the service as seeds (even the newly launched node). Seed nodes do not need to wait for other nodes to be ready. Hence, we can get around the requirement to launch one node at a time. Note we are only able to get around this requirement when launching nodes into a newly launched empty cluster. Once we have data in the cluster, we need to allow nodes to stream data from peers so they can start serving requests.

Specifying Cluster Topology

If you Execute Shell in one of the containers and run *nodetool status y*ou will see that all your nodes are in the same data center and more problematically in the same rack. If you are launching nodes across cloud providers or geographical regions, the latency will be such that your cluster will not be very performant. In these scenarios, you have to partition your nodes into data centers. Multi-data center clusters are able to provide low latency writes (and potentially low latency reads based on consistency requirements) across high latency network links because of asynchronous synchronization.

In addition to single data center, we are also using a single rack, which is more problematic. We mentioned earlier that Cassandra has configurable consistency guarantees. Essentially, you can specify how many copies of your data are written and read to service requests. If you split your cluster into racks, then copies of your data are spread out across racks. However, if you have just one rack then copies of the data are on arbitrary nodes. This means if you need to take nodes offline it is very difficult to decide which nodes to take offline without loosing ALL copies of SOME data. If you had racks, its always possible to take down N/2 racks without loosing any data. i.e. if you have 3 racks you can take one offline without data loss.

If you are using a cloud like AWS to run your containers, then it is important to create a separate rack for each Availability Zone (AZ). This is because Amazon guarantees that separate AZs have independent power sources, and network hardware etc. If there is an outage in one AZ it is likely to impact many nodes in that AZ. If your Cassandra cluster is not rack aware, you can loose multiple nodes and potentially all copies of some of your data. However, if you data is laid out in racks and all nodes in one AZ are not available you are still guaranteed to have redundant copies of your data.

To specify the Rack and Data Center, add the CASSANDRA_RACK and CASSANDRA_DC parameters into your environment. We will need to create separate services for each of the racks so that we can specify the properties separately. You will also need to set the CASSANDRA_ENDPOINT_SNITCH parameter to one of the snitches defined here. The default SimpleSnitch is not able to process racks and data centers but, the others such as the GossipingPropertyFileSnitch will be able to setup clusters which are data center and rack aware. For example, the docker-config.yaml below sets up a single data center (called aws-us-east) and three rack cluster. You can copy the CassandraX section to create more racks and data centers as needed. Note that you can use scale to increase the number of nodes in each of the racks as needed.

Cassandra1:
  environment:
    RANCHER_ENABLE: 'true'
    CASSANDRA_RACK: 'rack1'
    CASSANDRA_DC: aws-us-east
    CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: usman/docker-rancher-cassandra
  stdin_open: true
  volumes:
    - /var/lib/cassandra
Cassandra2:
  environment:
    RANCHER_ENABLE: 'true'
    CASSANDRA_RACK: 'rack2'
    CASSANDRA_DC: aws-us-east
    CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: usman/docker-rancher-cassandra
  stdin_open: true
  volumes:
    - /var/lib/cassandra
Cassandra3:
  environment:
    RANCHER_ENABLE: 'true'
    CASSANDRA_RACK: 'rack3'
    CASSANDRA_DC: aws-us-east
    CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: suman/docker-rancher-cassandra
  stdin_open: true
  volumes:
    - /var/lib/cassandra

Note that we still are limited by the requirement that only one node can be launched per Rancher service (Rack/DC). Also note that since we use the containers parent, Rancher service to determine seed nodes right now to each of your racks is disjointed from the others, essentially each rack is its own Cassandra cluster. In order to unify these into one cluster, we will need to use a shared set of seed nodes. In the next section, we will cover how to setup a separate shared seed service.

Seed Cassandra Service

So far, we have used the default Rancher Cassandra seed behavior, which means that every node will see all other nodes in its own Rancher service as seed nodes. If you ran the docker-compose.yaml file in the previous section, you will notice that your cluster is partitioned (i.e. none of the racks know about each other). This is because nodes only knows about seeds in its own Rancher service. Without shared seeds, the racks do not know about each other. To get around this limitation, we setup a separate seed service which will be used by all nodes in the rack. We then use the RANCHER_SEED_SERVICE property to specify that all members of this service will be considered seed nodes. A snippet of the updated docker-compose.yaml file is shown below. The complete file can be seen here.

CassandraSeed:
  environment:
    RANCHER_ENABLE: 'true'
    RANCHER_SEED_SERVICE: CassandraSeed
    CASSANDRA_RACK: 'rack1'
    CASSANDRA_DC: aws-us-east
    CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: usman/docker-rancher-cassandra
   stdin_open: true
   volumes:
     - /var/lib/cassandra
Cassandra1:
  environment:
    RANCHER_ENABLE: 'true'
    RANCHER_SEED_SERVICE: CassandraSeed
    CASSANDRA_RACK: 'rack1'
    ...

Cassandra2:
  environment:
    RANCHER_ENABLE: 'true'
    RANCHER_SEED_SERVICE: CassandraSeed
    CASSANDRA_RACK: 'rack1'
    ...

Note that now that all your Rancher services are in one cluster, you can only add one node to the entire cluster at a given time. Once you add a node, you have to wait for it to be bootstrapped by its peers before we can launch more nodes. This means that we can only launch one of the non-seed nodes at a given time.

Conclusion

Today, we have covered how to setup a data center and rack aware Cassandra cluster on top of Docker using Rancher. We saw that the Rancher/Docker service model does not fit cleanly with Cassandra’s seeding model and we had to use multiple services create a workable solution. In addition, Cassandra only allows us to add one new node at any given time which means we have to be careful about how we launch services. We have to launch the seed service before the rack services. We have to make sure we only scale up each rack service at any given time. In addition, we can only scale up one rack services at a time. This requirement would be familiar to anyone who runs Cassandra but is at odds with most services running on Docker and Rancher.

Furthermore, containers are designed to be disposable and come up and down often. With our Cassandra setup, we are assuming (and have to ensure) that our containers are launched, and scaled in a coordinated controlled fashion. For these reasons, running production database workloads in Docker and Rancher is still a risky proposition and should only be attempted if you are able to bring the commit database and infrastructure expertise to this part of your infrastructure.

To learn more about running other types of applications in Rancher, download our eBook: Continuous Integration and Deployment with Docker and Rancher.