Setting Up Shared Volumes with Convoy-NFS

Introduction

If you have been working with Docker for any length of time, you probably already know that shared volumes and data access across hosts is a tough problem. While the Docker ecosystem is maturing, implementing persistent storage across environments still seems to be a problem for most folks. Luckily, Rancher has been working on this problem and come up with a unique solution that addresses most of these issues. Running a database with shared storage still isn’t widely recommended, but for many other use cases, sharing volumes across hosts is good practice. Much of the guide was inspired by one of the Rancher Online meetups. Additionally, here is a little reference to go from that includes some of the NFS configuration information if you want to build something like this yourself from scratch.

Rancher Convoy

If you haven’t heard of it yet, the Convoy project by Rancher is aimed at making persistent volume storage easy. Convoy is a very appealing volume plugin because it offers a variety of different options. For example, there is EBS volume and S3 support, along with VFS/NFS support, giving users some great and flexible options for provisioning shared storage.

Dockerized-NFS

This is a little recipe for standing up a Dockerized NFS server for the convoy-nfs service to connect to. Docker-NFS is basically a poor man’s EFS, and you should only run this if you are confident that the server won’t get destroyed or the data simply isn’t important enough to matter if it is lost. You can find more information about the Docker NFS server I used here. To beef things up, I would suggest looking at the AWS implementation of NFS called Elastic File Storage or EFS. This solution is a much more robust and production-ready NFS server that you can use as the backend for Convoy-NFS. Setting up EFS is pretty simple, but out of scope for this article. Please check out the EFS docs on how to set up and configure EFS. One caveat: since EFS is a newer service to AWS, it is only available in a few locations (but there are more on the way soon) Here is what your docker-compose.yml might look like, for the docker-nfs server:

docker-nfs:
 image: cpuguy83/nfs-server
 privileged: true
 volumes:
 - /exports
 command:
 - /exports
 - /etc/services:/etc/services

One gotcha that you might encounter with this containerized NFS server method is the host either does not have the NFS kernel module installed, or doesn’t have the accompanying services turned on. On Ubuntu it is easy to install the kernel module. SSH to the host that will run the NFS server container and run the following command:

sudo apt-get install nfs-kernel-server

On CoreOS the module is installed, just not turned on. To enable NFS, you will need to SSH to the host that will run the NFS server container and run the following command:

sudo systemctl start rpc-mountd

Configuring EFS

Configuring EFS is really easy. Using the link posted above, you can go through the steps for creating an EFS volume for Convoy to connect to. When creating EFS volumes, either take note of the IP address of the EFS share, or if you drill into the configuration after creating the volume, the DNS name provided by AWS for the share. There is also a new catalog entry in the Rancher community catalog for using EFS volumes directly through Convoy, which can potentially simplify certain configurations. Using the catalog entry will still require you to create the EFS share first, but makes configuring Convoy to connect to it easier. Just copy the EFS ID out of AWS, choose the region in which you’ve created your share, and then specify where to mount the EFS share to locally. \“/efs\” is a good example to test things out initially:

Convoy-NFS

Currently, Rancher offers another catalog item called \“Convoy-NFS\” to connect containers to an NFS server: The setup is pretty straightforward, but there are a few things to note. First, the stack must be named \“convoy-nfs\“, which is the name of the plugin. Next, the NFS server should match the hostname on which you set up your NFS server; if you created the docker-nfs container instead, use the IP of the container (I used a test environment for this post, and thus just used the Rancher container internal IP address for the deployed NFS server). With EFS, use the DNS name configured when you created your NFS shares. The last thing to be aware of are the mount options and mount point. Match the port here with the port that the NFS server was configured with (2049 for docker-nfs) and make sure to turn on nfsver=4. Also be sure to use \“/\” for the MountDirectory if using the nfsvers=4 option; otherwise use \“/exports\“. The final configuration should look similar to the following.

proto=tcp,port=2049,nfsvers=4

You can add other options to tune the shares, but these are the necessary components for a bare minimum setup. Give Rancher a few minutes to provision the convoy-nfs containers. After everything turns green you should be able to create and attack NFS volumes. The fastest way to check if things are working is to click on the Infrastructure -> Storage Pools option. If you see hosts in that view, then you should be ready to start creating and sharing volumes. At this point you can either manually create a volume from the Storage Pools view, or simply create a service that uses the convoy-nfs driver and the volume name. I will create a testing container that spans across two hosts, that shares the same \“test_volume\” to share the data across hosts, which looks like the following (I used rancher-compose locally to spin up this test stack. Feel free to use the GUI if it is easier):

test:
 image: ubuntu
 volume_driver: convoy-nfs
 tty: true
 volumes:
 - test_volume:/data
 command:
 - bash

A new volume should pop up in the Storage Pools page: You can verify things are working by scaling up the number of containers to two or more. Then exec into one container, create a file in it and see if you can read the file from another container, preferably on a different host. If that all works, you should be all set: From the first container, we can write out a file: And from the second container, we can read it back to verify the shared storage is working:

Conclusion

As Convoy continues to evolve and grow, we should see a lot more options for deployments. For now, NFS is stable and just works; using EFS makes things even easier if you are already on AWS. I’m looking forward to seeing what is next for Convoy and for shared volumes in Rancher. Josh Reichardt is a DevOps engineer at about.me, where he builds and maintains their infrastructure, among other things. You can visit his blog at thepracticalsysadmin.com to find more interesting DevOps-related content and on Twitter (@Practical_SA).