关注微信公众号
第一手干货与资讯
加入官方微信群
获取免费技术支持
In the third section on data resiliency, we delve into various ways that data can be managed on Rancher (you can catch up on Part 1 and Part 2 here). We left off last time after setting up loadbalancers, health checks and multi-container applications for our WordPress setup. Our containers spin up and down in response to health checks, and we are able to run the same code that works on our desktops in production. Rancher Multi-container Wordpress All of this is nicely defined in a docker-compose.yml file along with the rancher-compose.yml companion that extends compose’s functionality on the Rancher cluster. The only issue is that when we terminated the MySQL container all of the data was lost. Browsing through the Docker documentation on layers, we can see the following diagram from the docker documentation images and containers. Container layers A container is composed of a read-only bundle of layers that is built from the image, followed by a thin read-write layer that is called the container layer. The Docker storage driver is responsible for stacking these layers and providing a single unified view.
When the container is deleted, the writable layer is also deleted. The underlying image remains unchanged.
So after deleting the MySQL containers, we had to setup Wordpress from the beginning, and to make matters worse, all the posts that we wrote had been lost. How do we prevent this? What options do I have to do this?
In general, the top level choices are the following:
1. Offload the stateful MySql to a PaaS like RDS 2. Mount the MySQL data volume onto a host
If you picked option 1, then that means you have decided to skip the trouble of working with data management inside containers; this frees your team to focus on building the stateless parts of your product. The Rancher UI makes prototyping and working with a team on applications a breeze, so you can just point the applications at a database URL and let an external vendor handle it. Again, a big portion of having a reliable system is operational knowledge, if there is minimal benefit to spending man hours on it, then offloading the burden a trusted solution will let you use Rancher in production effectively faster. (This is similar to the first post, where we chose RDS to host our Rancher DB in HA mode). Though if you are curious, or have the man hours to spare, option 2 becomes quite enticing. The benefits of this is total end user control and potential cost savings, as we avoid paying the vendor’s markup. Regardless of which choice we end up making, getting to know more about Docker volumes is a benefit for either choice. From here on, we will dig into container data management. At the end of the post, we hope you have all the knowledge and resources to make the best choice for your Rancher project. The most simple case is to modify the wordpress-multi docker-compose.yml from our last post on health checks and networking, we will mount a volume on the host’s filesystem. Adding these lines to the volumes: - /my/own/datadir:/var/lib/mysql to our docker-compose.yml:
volumes: - /my/own/datadir:/var/lib/mysql
version: '2' services: mywordpress: tty: true image: wordpress links: - database:mysql stdin_open: true wordpresslb: ports: - 80:80 tty: true image: rancher/load-balancer-service links: - mywordpress:mywordpress stdin_open: true database: environment: MYSQL_ROOT_PASSWORD: pass1 tty: true image: mysql # New volumes mounted to the host drive! volumes: - /my/own/datadir:/var/lib/mysql stdin_open: true
The above is the simplest data management solution to use. If we read through the documentation posted on the official MySQL container. It contains a detailed documentation on managing data sectioned Where to Store Data. Most official database containers have these steps detailed in Dockerfiles, but in general there are 2 main types of Docker data management: 1. Data volumes 2. Data volume containers A data volume is created by Docker when we define it in the run command, create command, or in the Dockerfile. For the case of the MySQL image, the anonymous volume is defined in the MySQL container’s Dockerfile as the volume /var/lib/mysql.
# Mysql dockerfile snippet ... VOLUME /var/lib/mysql COPY docker-entrypoint.sh /usr/local/bin/ RUN ln -s usr/local/bin/docker-entrypoint.sh /entrypoint.sh # backwards compat ENTRYPOINT ["docker-entrypoint.sh"] EXPOSE 3306 CMD ["mysqld"]
So actually when we deleted our MySQL container, we didn’t lose the data per-say. It is just that the new container created a new data volume, and we lost reference to the old one.
The *Volume* flag in a Dockerfile, produces the same results as running a container with the -v option. e.g. \“docker run -v /var/lib/mysql ubuntu bash\”
docker inspect r-wordpress-multi_database_1 ... "Mounts": [ { "Name": "79f02c7ed4e7bd3b614f4f19d6b121125640a0ec5ebf873811b58a86c4faad62", "Source": "/var/lib/docker/volumes/79f02c7ed4e7bd3b614f4f19d6b121125640a0ec5ebf873811b58a86c4faad62/_data", "Destination": "/var/lib/mysql", "Driver": "local", "Mode": "", "RW": true, "Propagation": "" } ],
The folder is mounted on our system under the docker volumes at \“/var/lib/docker/volumes/79f02c7ed4e7bd3b614f4f19d6b121125640a0ec5ebf873811b58a86c4faad62/_data\“. This is the default and is easy and fairly transparent to the user. The downside is that the files may be hard to locate for tools and applications that run directly on the host system, i.e. outside containers.
$> ls -ltr /var/lib/docker/volumes/79f02c7ed4e7bd3b614f4f19d6b121125640a0ec5ebf873811b58a86c4faad62/_data total 188448 -rw-r----- 1 999 docker 50331648 Sep 6 01:17 ib_logfile1 -rw-r----- 1 999 docker 56 Sep 6 01:17 auto.cnf drwxr-x--- 2 999 docker 4096 Sep 6 01:17 performance_schema drwxr-x--- 2 999 docker 4096 Sep 6 01:17 mysql drwxr-x--- 2 999 docker 12288 Sep 6 01:17 sys drwxr-x--- 2 999 docker 4096 Sep 6 01:17 wordpress -rw-r----- 1 999 docker 426 Sep 25 14:38 ib_buffer_pool -rw-r----- 1 999 docker 79691776 Oct 22 16:20 ibdata1 -rw-r----- 1 999 docker 50331648 Oct 22 16:20 ib_logfile0 -rw-r----- 1 999 docker 12582912 Oct 25 23:33 ibtmp1
The caveat of an anonymous volume is that Docker will not re-attach the data volume when a new container is started; when the new MySQL container restarted it creates a new Docker Volume instead of attaching our previous volume.
You will also have dangling volumes that take up disk space when you don’t perform cleanup regularly.
$> docker inspect --format '{{ range .Mounts}}{{ .Source }}{{ end }}' r-wordpress-multi_database_1 /var/lib/docker/volumes/79f02c7ed4e7bd3b614f4f19d6b121125640a0ec5ebf873811b58a86c4faad62/_data $> docker rm -f r-wordpress-multi_database_1 $> # Rancher spins up a new database $> docker inspect --format '{{ range .Mounts}}{{ .Source }}{{ end }}' r-wordpress-multi_database_1 /var/lib/docker/volumes/bbb24f2878a01f068cdaaa66ad3461996c48582cbd65166db2e10f90c03c1918/_data
A way to ensure that the data volume follows the container is link it to a folder on the host machine. In this case, all we need to do on a docker command is to add a target on the host system.
$> docker run --name some-mysql -v /my/own/datadir:/var/lib/mysql -d mysql
When we mount a host directory onto Docker’s union file system, it does not copy any of the existing files from lower layers. The -v /my/own/datadir:/var/lib/mysql syntax will replace files in the layers below it, Whereas an anonymous volume created with -v /somefolder will container data from any files in /somefolder from the lower read-only layers.
The previous docker command is represented in docker-compose below:
... database: environment: MYSQL_ROOT_PASSWORD: pass1 tty: true image: mysql # New volumes mounted to the host drive! volumes: - /my/own/datadir:/var/lib/mysql stdin_open: true
Since this data volume mount is named, we can terminate the database container and start a new one with the same data volume mount and retain our data. Of course, this is a simplification of the actual MySQL container. The actual layer images would look more like the following diagram: union file system with host mount
Do note the read-write layer still takes up disk space in /var/lib/docker, and be mindful of limited root disk sizes on public clouds like AWS. Placing the Docker folders on a separately mounted drive prevents Docker from eating up the host OS’s disk space.
The extension to this the data volume container. A data volume container is just as it sounds, it is a container that is responsible for holding a reference to data volumes.
In fact, this is exactly how upgrading Rancher versions works, in the Rancher upgrade documentation.
docker create --volumes-from \ --name rancher-data rancher/server:
Now you see that we have started the new container with --volumes-from rancher-data. The --volumes-from command tells the create command to just reference the volumes from the existing Rancher server container instead of creating new volumes. This reference ensures that the volumes of the old Rancher container are still around, even if we kill the original container. The newly created data volume container is now holding a reference to the original MySQL volumes of your old Rancher server. Rancher Server creates the following volumes:
VOLUME /var/lib/mysql /var/log/mysql /var/lib/cattle
Creating data volumes results in the following image on the host. Do note that the drive paths are inside the Docker folder, so these are anonymous volumes. host mount data volume Since Rancher server runs MySQL, this approach can be applied to our own application. So this time we can modify our docker-compose.yml to create a Docker data container for managing the MySQL data.
... databaseData: image: mysql entrypoint: /bin/bash database: environment: MYSQL_ROOT_PASSWORD: pass1 tty: true image:mysql # New volumes mounted to the host drive! volumes_from: - databaseData stdin_open: true
With a Docker data container, we can even share the volumes with multiple other containers. So long as you don’t perform docker-compose rm -v, the databaseData container will not lose the data volume, and you can cycle the database container as often as you would like. So now we have a setup with data containers that persist our MySQL data. The issue now is that we need to tie our MySQLs to a specific set of hosts on our Rancher environment. This is fine for containers scheduled to run on dedicated hardware, but it requires a degree of manual control. What happens if we have multiple hosts, and the MySQL container gets scheduled to a different host than the data container?
A solution is to use Rancher Sidekicks. It was specifically designed to allow the scheduling of secondary (data container) services that must be deployed on the same host. You can read more about sidekicks on the documentation, but the simple case is to add the io.rancher.sidekicks: label to the primary container like so:
... databaseData: image: mysql entrypoint: /bin/bash database: environment: MYSQL_ROOT_PASSWORD: pass1 tty: true image:mysql # New volumes mounted to the host drive! volumes_from: - databaseData stdin_open: true labels: io.rancher.sidekicks: databaseData
This will ensure that the data container will be scheduled along with the parent container on the same host. With what we covered so far, reading complicated catalogs like such as the ElasticSearch Rancher Catalog with various sidekick containers and data volumes will now be a lot easier to grok.
Building off the concept of data volumes, Docker provides a feature called volume drivers, specifically the Rancher Lab’s Convoy. A volume driver is a Docker plugin that extends the data volume functionality of the Docker engine. Specifically, Docker Engine volume plugins enable Engine deployments to be integrated with external storage systems, such as Amazon EBS, and enable data volumes to persist beyond the lifetime of a single Engine host. You can read more about that in the volume plugin documentation. In essence, what the Docker Volume driver like Convoy does is add custom behavior inside hooks during the volume creation process. For example, we can create a volume based off of external storage, such as S3 or a remote NFS. With a Convoy setup in place, we can have our MySQL container data storage backed up to a remote location and then recreated on another host if needed. All of this external integration is managed through the Convoy API. Any volume designated to specific containers created through convoy will be stored based on driver configurations. For the next section, we reference an existing post about Convoy-NFS on Rancher.
To setup Convoy, we can follow the instructions from the official Convoy installation guide. Do note that since Convoy is a Docker plugin, we will need to provision it on every one of our Rancher hosts. The Rancher team has provided a nice Convoy-NFS catalog item to do it for us, but here we will work through the process of setting Convoy up manually just to learn how it operates.
wget https://github.com/rancher/convoy/releases/download/v0.5.0/convoy.tar.gz tar xvf convoy.tar.gz sudo cp convoy/convoy convoy/convoy-pdata_tools /usr/local/bin/ sudo mkdir -p /etc/docker/plugins/ sudo bash -c 'echo "unix:///var/run/convoy/convoy.sock" > /etc/docker/plugins/convoy.spec'
Next we then setup a NFS system and connect our Convoy system to it. We use a prototype version of the NFS system and start it up.
There is a host kernel requirement to docker-nfs which requires some extra setup time. You may need to install nfs-common and nfs-kernel-server package on ubuntu or similar for your OS or VM.
$> docker run -d --name nfs --privileged -v /convoy-data:/convoy-data codesheppard/nfs-server /convoy-data $> docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3b39b9313ed8 codesheppard/nfs-server "/usr/local/bin/nfs_s" About a minute ago Up 56 seconds 111/udp, 2049/tcp nfs ... $> docker inspect --format '{{ .NetworkSettings.IPAddress }}' nfs 172.17.0.7
Then we now have a NFS server listening on 172.17.0.7, the next step is to connect the Convoy daemon to it. We reference the mounting that is done in the rancher/convoy-agent project (supports Rancher’s Convoy-NFS catalog) and setup documentation from rancher/wiki.
sudo mkdir /nfs sudo mount -t nfs -o nolock 172.17.0.7:/convoy-data /nfs sudo convoy daemon --drivers vfs --driver-opts vfs.path=/nfs ... misc convoy driver logs ...
Convoy daemon allows multiple drivers to be run
--drivers [--drivers option --drivers option]Drivers to be enabled, first driver in the list would be treated as default driver
--driver-opts [--driver-opts option --driver-opts option] options for driver
Our host’s Docker engine now has a plugin to run Convoy. So lets assign our MySQL data to convoy engine through docker-compose.yml.
... database: environment: MYSQL_ROOT_PASSWORD: pass1 tty: true image:mysql # New volumes mounted to the host drive! stdin_open: true volume_driver: convoy volumes: - 'wordpress_mysql:/var/lib/mysql' rancher-compose up --upgrade
Once we hit the upgrade button, we can then see the Convoy driver setting up the new drive in the logs.
... DEBU[0037] Volume: wordpress_mysql is mounted at for docker pkg=daemon DEBU[0037] Response: {} pkg=daemon DEBU[0037] Handle plugin mount volume: POST /VolumeDriver.Mount pkg=daemon DEBU[0037] Request from docker: &{wordpress_mysql map[]} pkg=daemon DEBU[0037] Mount volume: wordpress_mysql for docker pkg=daemon DEBU[0037] event=mount object=volume opts=map[MountPoint:] pkg=daemon reason=prepare volume=wordpress_mysql DEBU[0037] event=list mountpoint=/nfs/wordpress_mysql object=volume pkg=daemon reason=complete volume=wordpress_mysql DEBU[0037] Response: { "Mountpoint": "/nfs/wordpress_mysql" } pkg=daemon ...
If you follow along on your NFS mounted drive, you can see that the mounted NFS directory now has 2 folders, a config and wordpress_mysql folder. There are now various functions that can now be done, but we can now run experiments on Convoy with our simple setup! Convoy supports various commands, but a quick dump of the CLI help should be a great starting point to browse. For more information, the offical Convoy documentation is the best place to get started. Some of the following commands of interest are:
# Create and Delete sudo convoy create volume_name sudo convoy delete # Snapshot creation sudo convoy snapshot create vol1 --name snap1vol1 # Backup snapshot sudo convoy backup create snap1vol1 --dest s3://backup-bucket@us-west-2/ sudo convoy create res1 --backup
Experimenting with the Convoy driver, we can create volumes and delete them. A most useful feature is a nice S3 backup. Conceptually, we are still mounting a directory to the container, but now it is backed by a network drive. We can now inspect the container and check the differences.
$> docker inspect "HostConfig": { "Binds": [ "wordpress_mysql:/var/lib/mysql:rw" ], "VolumeDriver": "convoy", ... "Mounts": [ { "Name": "wordpress_mysql", "Source": "/nfs/wordpress_mysql", "Destination": "/var/lib/mysql", "Driver": "convoy", "Mode": "rw", "RW": true, "Propagation": "rprivate" } ...
If you check the drive that we are sharing through our NFS setup, we can see that the /var/lib/mysql data is mounted on our NFS drive!
$> ls -ltr /nfs/wordpress_mysql total 188448 -rw-r----- 1 999 docker 50331648 Sep 6 01:17 ib_logfile1 -rw-r----- 1 999 docker 56 Sep 6 01:17 auto.cnf drwxr-x--- 2 999 docker 4096 Sep 6 01:17 performance_schema drwxr-x--- 2 999 docker 4096 Sep 6 01:17 mysql drwxr-x--- 2 999 docker 12288 Sep 6 01:17 sys drwxr-x--- 2 999 docker 4096 Sep 6 01:17 wordpress -rw-r----- 1 999 docker 426 Sep 25 14:38 ib_buffer_pool -rw-r----- 1 999 docker 79691776 Oct 22 16:20 ibdata1 -rw-r----- 1 999 docker 50331648 Oct 22 16:20 ib_logfile0 -rw-r----- 1 999 docker 12582912 Oct 25 23:33 ibtmp1
Now after testing it out, a manual Convoy setup has lots of moving parts. So from now, we can look towards the community and Rancher core team to help expediate the process of using Convoy.
With our toy NFS, the next step would be to drop the manual Convoy setup, and connect the official Rancher Convoy-NFS catalog to it. So with our simple setup and understanding of the NFS + Convoy system, we can now take a look at our reference blog post Convoy-NFS on Rancher, and apply our Convoy-nfs catalog to our Rancher setup!
The setup is pretty straightforward, but there are a few things to note. First, the stack must be named \“convoy-nfs\“, which is the name of the plugin. Next, the NFS server should match the hostname on which you set up your NFS server; if you created the docker-nfs container instead, use the IP of the container 172.17.0.7.
The last thing to be aware of are the mount options and mount point. Match the port here with the port that the NFS server was configured with (2049 for docker-nfs) and make sure to turn on nfsver=4. Also be sure to use \“/\“? for the MountDirectory if using the nfsvers=4 option; otherwise use \“/exports\“.
proto=tcp,port=2049,nfsvers=4
The final configuration should look similar to the following:
You can add other options to tune the shares, but these are the necessary components for a bare minimum setup.
For the time being I will continue to utilize the same locally hosted NFS server. Though once you have a real NFS or EFS server, we only need to mount it to a standardized drive, and all our Rancher nodes will have access to a shared network drive with backup capabilities.
Also note, that there is a community supported Convoy-EFS Catalog that supports an AWS EFS. There are some vendor specific differences, but in essence EFS is a managed NFS setup. So now we watch our stack spin up on Rancher here. Rancher Convoy-EFS You will now see that Rancher is starting up its storage pool. This opens up the next section on the Rancher UI that shows the storage volumes. Go to Infrastructure > Storage on the menu, and you should see the following view for volumes managed through Rancher. Rancher Storage Pool After we fill in the details on the catalog convoy we modify our *docker-compose.yml* to match.
For this catalog to work the volume_driver must be called convoy-nfs.
... database: environment: MYSQL_ROOT_PASSWORD: pass1 tty: true image:mysql # New volumes mounted to the host drive! volume_driver: convoy-nfs volumes: - mysql1:/var/lib/mysql stdin_open: true rancher-compose up --upgrade
You can browse through the logs from the UI on the convoy-nfs agent service to follow along.
Then after the upgrade, you will see the same volumes on the mounted volume on our NFS shared drive with Convoy agent managed through Rancher. We now have a full setup with the certified Rancher NFS catalog. Final Setup With the final setup, we can now take a look at the various features of Convoy. Lets look at the volumes that were created on the UI. Seeing volumes We can also see this with the command line tool:
$> sudo convoy -s /var/run/convoy-convoy-nfs.sock list { "mysql1": { "Name": "mysql1", "Driver": "vfs", "MountPoint": "/var/lib/rancher/convoy/convoy-nfs-f3f0877f-1d72-4902-b99b-a745646e1e37/mnt/mysql1", "CreatedTime": "Tue Nov 08 16:22:26 +0000 2016", "DriverInfo": { "Driver": "vfs", "MountPoint": "/var/lib/rancher/convoy/convoy-nfs-f3f0877f-1d72-4902-b99b-a745646e1e37/mnt/mysql1", "Path": "/var/lib/rancher/convoy/convoy-nfs-f3f0877f-1d72-4902-b99b-a745646e1e37/mnt/mysql1", "PrepareForVM": "false", "Size": "0", "VolumeCreatedAt": "Tue Nov 08 16:22:26 +0000 2016", "VolumeName": "mysql1" }, "Snapshots": {} }, "mysqltest": { "Name": "mysqltest", "Driver": "vfs", "MountPoint": "", "CreatedTime": "Wed Nov 09 06:35:21 +0000 2016", "DriverInfo": { "Driver": "vfs", "MountPoint": "", "Path": "/var/lib/rancher/convoy/convoy-nfs-f369e5a5-deb7-4a9c-9812-845bd70dbecd/mnt/mysqltest", "PrepareForVM": "false", "Size": "0", "VolumeCreatedAt": "Wed Nov 09 06:35:21 +0000 2016", "VolumeName": "mysqltest" }, "Snapshots": {} } }
Running some experimentations, we can test some backup functionalities offered by Convoy. We first create a snapshot.
$> sudo convoy -s /var/run/convoy-convoy-nfs.sock snapshot create mysql1 snapshot-fec0417c7db1422d
Then we create a backup to S3:
$> sudo convoy -s /var/run/convoy-convoy-nfs.sock backup create snapshot-fec0417c7db1422d --dest s3://backup-bucket@us-west-2/ s3://backup-bucket@us-west-2/?backup=backup-a760a7f5338a4751\u0026volume=mysql1
> You may need to put credentials to /root/.aws/credentials or setup sudo environment variables in order to get S3 credential works. Or setup the credentials on the convoy-agent service.
$> sudo convoy -s /var/run/convoy-convoy-nfs.sock create restoredMysql --backup s3://backup-bucket@us-west-2/?backup=backup-a760a7f5338a4751\u0026volume=mysql1
Then we can now attach the data volume as *restoredMysql* to any container on our stack.
Now if you are on a cloud provider, we can use the networked block store volumes on the cloud in place of or in addition to NFS. You can have the hosts attach an EBS volume and ensure that the new EBS volumes are available to the containers. A similar process can be done on Azure and DigitalOcean. For AWS, I setup my Rancher Agent nodes with the following userdata, this installs Convoy on my agent nodes on startup.
#!/bin/bash yum install docker -y service docker start usermod -a -G docker ec2-user # Download convoy onto our hosts wget https://github.com/rancher/convoy/releases/download/v0.5.0/convoy.tar.gz tar xvf convoy.tar.gz cp convoy/convoy convoy/convoy-pdata_tools /usr/local/bin/ mkdir -p /etc/docker/plugins/ bash -c 'echo "unix:///var/run/convoy/convoy.sock" > /etc/docker/plugins/convoy.spec' # Bootstrap for Rancher agent docker run -e CATTLE_HOST_LABELS='foo=bar' -d --privileged \ -v /var/run/docker.sock:/var/run/docker.sock rancher/agent:v0.8.2 \ http://:8080/v1/projects/1a5/scripts/
You can daemonize the Convoy process for more hands-off setup. Your hosts must also be configured to run EBS volumes, so we can quickly do that by setting up an IAM Instance Profile for our instances. An Instance Profile is essentially a way to use the AWS API from within your EC2 instances without having to manage the API key manually. You can read more about it on the AWS documentation, but essentially to use convoy with EBS you must provide and IAM Instance Profile with the following list of permissions, as described on the Convoy EBS documentation. You can just use the EC2 PowerUser IAM policy for a quickstart. Specific policy tunings will just require launching a new host.
"ec2:CreateSnapshot", "ec2:CreateTags", "ec2:CreateVolume", "ec2:DeleteVolume", "ec2:AttachVolume", "ec2:DetachVolume", "ec2:DescribeSnapshots", "ec2:DescribeTags", "ec2:DescribeVolumes"
When we save a volume backed by EBS, we are actually saving it as an EBS image which can be mounted on another host. One caveat is that the current host must be in the same region of backup to be restored. Once our node is up, we will need to go in and start Convoy. Preferably, the convoy daemon is started inside a process manager by supervisor or upstart, otherwise it runs in the foreground by default.
sude convoy daemon --drivers ebs
With our daemon setup, we can now run some catalogs to test out how Convoy functions. Lets use rancher-compose to launch the following stack.
postgres: image: postgres ports: - 5432 volumes: - db_vol:/var/lib/postgresql/data volume_driver: convoy
From here on out, you can test out the various Convoy commands from within the instance to test out the various features that allow you to dynamically attach and detach EBS volumes from within your instance. Say for example, the EC2 instance was lost: the EBS volume will be marked as available. Transferring this available EBS to another EC2 instance with Docker and reschedulling the container is fairly easy. If we know the volume id, we can use the following Docker command to remount the EBS onto another EC2 host.
docker volume create --driver convoy --opt driver=ebs --opt id=vol-12345 --name myvolume
Lastly, if we want to destroy the EBS volumes so we don’t get charged for unused volumes:
# remove stack and volumes rancher-compose rm -v
We have now performed a brief walkthrough of the data resiliency solutions offered by Rancher and Docker. This post is quite long and involved, but since handling data in clustered environments is a pretty complicated topic, this is actually considered too short. In this we hope that the content here will serve as a basis to get started playing around with the existing solutions to find what is the best fit for your project. Thanks for reading! Nick Ma is an Infrastructure Engineer who blogs about Rancher and Open Source. You can visit Nick’s blog, CodeSheppard.com, to catch up on practical guides for keeping your services sane and reliable with open-source solutions.