关注微信公众号
第一手干货与资讯
加入官方微信群
获取免费技术支持
Hi, I am Sheng Yang (@yasker), an engineer here at Rancher Labs. Over the last few months our team has been working on building Docker storage software that would allow users to create and manage persistent Docker volumes. With last week’s release of Docker 1.8, which now officially supports Docker volume drivers, I am excited to announce Convoy, an open-source Docker volume driver that can snapshot, backup and restore Docker volumes anywhere. At Rancher Labs, we are building an infrastructure service layer that offers uniform and consistent storage and networking services across any cloud provider, virtualized data center, or bare metal servers. Our overlay networking solution has been available for some time, and making persistent storage as portable as Docker containers is our next important goal. We wanted to develop a solution that was as simple to use and portable as Docker itself.
Convoy is designed to be standalone Docker storage services. Specifically, it is a Docker volume driver that runs on individual Linux hosts. Our initial implementation of Convoy utilizes Linux Device Mapper to deliver four key Docker storage functions for volumes:
With this functionality, users that require persistent storage for Docker workloads can now snapshot, backup and restore data volumes anywhere. During the Rancher Beta, we’ve talked to many users who are looking to run databases in Dockerized environments, or maintain stateful logs or indexes on a host, often for testing, DR or compliance reasons. With Convoy these users have a simple tool for maintaining state in Docker volumes. Convoy is a light-weight Go program that developers can script to take snapshots or perform backups of individual container volumes. You can download the binary on GitHub today. At this point Convoy is available as a standalone program, and has not yet been integrated into Rancher. If you’re using Rancher you should see Convoy support arriving sometime in the next few releases. Convoy is a Docker Volume Driver and connects with different storage platforms
We built Convoy to be an extensible framework that would simplify implementing Docker storage plugins while supporting vendor-specific extensions such as QoS, snapshots, replication, backup, and restore. Convoy can support vendor-specific extensions for any storage backend by writing Convoy Drivers. We are launching the project with the Linux Device Mapper driver, as well as a VFS/NFS driver. All drivers support basic volume operations like create, delete, mount, unmount, and list. A Convoy driver can easily add support for snapshots, backup and restore functionality. If you’re interested in adding additional capabilities to a Convoy driver or building new Convoy drivers for additional storage backends, please submit an issue or a pull request on GitHub, and our team will be happy to help or review your work. We are also working on integrating Convoy with Kubernetes volume driver functionality, so that Convoy drivers can be utilized within both Docker and Kubernetes orchestration frameworks.
Installing Convoy on a Linux host is straightforward; simply download the binary tar ball and launch the Convoy CLI to get started. Our instructions in GitHub will then walk you through setting up and starting the Convoy daemon and configuring local block devices from the host’s local disks. Once the block devices are created you can begin creating docker volumes.
Convoy is a Docker volume plugin, which means you can create and destroy volumes directly from the Docker CLI or API. To create a Convoy volume from the Docker CLI execute a command such as:
sudo docker run -it -v volume_name:/volume --volume-driver=convoy ubuntu
The name of the volume can be specified, however the size of the volume is based on default configurations within Convoy. Once a volume is created, it can be mounted by any container running on the host. Convoy by default creates a 100GB, thin provisioned volume with the Device Mapper driver. If you want to specify an arbitrary size of the Docker volume, you will need to use the Convoy CLI to create the volume.
To create a snapshot for a Convoy volume, you will need to use the Convoy CLI. To create a snapshot, users issue the following command:
convoy snapshot create (volume name or UUID) --name (snapshot name)
This process will take a complete snapshot of the volume locally and save it in the current host. Snapshots can be scheduled using cron or any external script. Snapshots are crash-consistent and are taken without the knowledge of the application. If your application supports application-consistent snapshots, you can run the appropriate commands using docker exec to quiesce the application.
cron
docker exec
Convoy currently supports backing up snapshots up to Amazon S3 or VFS/NFS, which make it easy and simple to migrate the volume across the network. To create a backup, you would issue a command such as:
convoy backup create (snapshot name or UUID) --dest s3://(bucket_name@region_name)/
Convoy then prints out the URL of the backup after the backup operation has completed successfully. We’ve utilized device mapper thin-provisioning mechanism to support delta block backup. This means, every time you backup a snapshot, we only backup the changed blocks of that snapshot to the object store. Convoy automatically figures out if additional backups need to be taken for previous snapshots so there is no need to maintain the chain of backups yourself. You can delete any backups you no longer want, and Convoy will automatically preserve the object store data set that is still in use by other backups. Convoy automatically garbage collects unused data sets in object storage.
Once you create a backup in the object store, you can recreate it in any host with Convoy installed by asking Convoy to create a new volume from the restored image:
convoy create (restored-volume-name) --backup <backup_url>
This allows users to backup a production volume and recreate it in a test environment, for example. Or snapshot a volume running in the cloud, and restore it on premise, or in another cloud provider.
The following video demonstrates the basic backup and restore operations.
Convoy is currently released as technology preview code. We’re still working on it, and would like to gather feedback from you. Please give it a tryand submit bugs and enhancement requests on github, or in the Rancher Forums. To learn more about Convoy, pleaseview a recordingof our latest online meetup where we demonstrated how to get started with Convoy.