Rancher Convoy adds Amazon EBS support with 0.3 release

We just released Convoy v0.3 last week, and I’m excited to announce that it now supports Amazon Elastic Block Store (EBS) as a Convoy driver. With this release you can now create persistent Docker volumes on AWS, backed with all the performance and features of EBS. With this new feature, when users create a Convoy volume using the EBS driver, Convoy will create an EBS volume, attach it to the current running instance, and then assign it to the Docker container. Convoy can also take snapshots of the volume and back it up to S3, as well as create a new volume from the backup. The snapshot and backup operations are implemented using the native EBS snapshot functionality. Further more, Convoy can take an existing EBS volume and use it to create a volume attached to a Docker container.

Configuring Convoy for EBS

You can find detailed instructions for installing convoy in my last post, or in Github. Once you have Convoy installed on a linux insrtance in EC2, make sure you’ve configured your AWS credentials on the host, and then start the Convoy daemon using the EBS driver:

sudo convoy daemon --driver ebs

Make sure you’re running on an EC2 instance, otherwise the Convoy daemon will fail to start. Notice: if the server has run Convoy before, Convoy will start with the previous configuration. You can update the convoy.cfg to use the new EBS driver, by executing the following command before starting Convoy daemon.

sudo cp /var/lib/convoy/convoy.cfg ~/convoy.cfg.bak
sudo sed -i 's/"DriverList":\[\(.*\)\]/"DriverList":\[\1,"ebs"\]/' /var/lib/convoy/convoy.cfg

At any time, you can check convoy info to see which driver you are running, and the configuration of the driver:

$ sudo convoy info
{
     "General": {
     "Root": "/var/lib/convoy",
     "DriverList": [
          "ebs"
     ],
     "DefaultDriver": "ebs"
     },
     "ebs": {
          "AvailablityZone": "us-west-2a",
          "DefaultVolumeSize": "4294967296",
          "DefaultVolumeType": "gp2",
          "InstanceID": “i-4caac789",
          "Region": "us-west-2"
     }
}

Creating an Docker Volume with EBS

To create a volume, you can use issue the following command on either the Convoy CLI or the Docker CLI. If you are using the Convoy CLI, where \“vol1\” is the name of the volume:

sudo convoy create vol1

Or, if you are using the Docker CLI:

sudo docker run -it -v vol1:/vol1 --volume-driver=convoy ubuntu

Either of these will create an EBS volume based on the options that are specified in the configuration file. By default, these options set the size of the volume at 4GB, the type of the EBS volume to gp2 . The region and availability zone where the volume is created will always be the same as the location of the EC2 Instance. Note: If you see following error:

"Error": "Error response from server, AWS Error:  NoCredentialProviders no valid providers in chain"

Please check your AWS credentials. By issuing an inspect command, you can view all of the details about your new EBS volume:

$ sudo convoy inspect vol1
{
    "UUID": "a0831463-e299-463c-8584-0da393623a69",
    "Name": "vol1",
    "Driver": "ebs",
    "MountPoint": "",
    "CreatedTime": "Thu Sep 10 15:10:39 -0700 2015",
    "DriverInfo": {
        "AvailablityZone": "us-west-2a",
        "CreatedTime": "Thu Sep 10 22:10:24 +0000 2015",
        "Device": "/dev/xvdf",
        "Driver": "ebs",
        "EBSVolumeID": "vol-ec44e118",
        "IOPS": "12",
        "MountPoint": "",
        "Size": "4294967296",
        "State": "in-use",
        "Type": "gp2",
        "UUID": "a0831463-e299-463c-8584-0da393623a69"
    },
    "Snapshots": {}
}

You can modify the create command with parameters to describe the size, type and IOPS for the volume. (You can only specify IOPS parameters for volumes of type io1 ). For example, If you want to create a 10GB volume of type io1, with 300 IOPS, you would use the following command:

sudo convoy create vol1 --type io1 --size 10G --iops 300

You can also create a docker volume from an existing EBS volume that is located in the same Availability Zone as the EC2 instance, by specifying the EBS volume ID. For example:

sudo convoy create vol1 --id vol-1234abcd

Snapshots, Backup and Recovery

To take a snapshot of the volume you’ve created, use the snapshot create command. You can optionally provide a name for the snapshot, as I have in this example:

sudo convoy snapshot create vol1 --name snap1vol1

When you issue the snapshot command, Convoy calls the AWS API and starts the process of creating a snapshot of the selected Docker volume. Amazon automatically backs up the snapshot to S3. This processes runs remotely, and you can inspect the snapshot at any time:

$ sudo convoy inspect snap1vol1
{
    "UUID": "0341d802-15fb-4318-a947-2e1a0acdbaac",
    "VolumeUUID": "a0831463-e299-463c-8584-0da393623a69",
    "VolumeName": "vol1",
    "VolumeCreatedAt": "Thu Sep 10 15:10:39 -0700 2015",
    "Name": "snap1vol1",
    "CreatedTime": "Thu Sep 10 15:11:50 -0700 2015",
    "DriverInfo": {
        "Driver": "ebs",
        "EBSSnapshotID": "snap-53161202",
        "EBSVolumeID": "vol-ec44e118",
        "Size": "4294967296",
        "StartTime": "Thu Sep 10 22:11:57 +0000 2015",
        "State": "pending",
        "UUID": "0341d802-15fb-4318-a947-2e1a0acdbaac",
        "VolumeUUID": "a0831463-e299-463c-8584-0da393623a69"
    }
}

Since the snapshot already backs up the volume to S3, you can use the Convoy backup command to retrieve the URL for the backup.

$ sudo convoy backup create snap1vol1
ebs://us-west-2/snap-53161202

Once you have the backup URL, ebs://us-west-2/snap-53161202 in this case, you can use the inspect command to get the details of the backup.

$ sudo convoy backup inspect ebs://us-west-2/snap-53161202
{
    "EBSSnapshotID": "snap-53161202",
    "EBSVolumeID": "vol-ec44e118",
    "Region": "us-west-2",
    "Size": "4294967296",
    "StartTime": "Thu Sep 10 22:11:57 +0000 2015",
    "State": "completed"
}

Finally we can restore the backup on any EC2 instance running Convoy in the same AWS region using the following command:

sudo convoy create res-vol1 --backup ebs://us-west-2/snap-53161202

That’s it! You can download Convoy from GitHub, and find more details about the EBS Driver here. If you’d like to watch a recent online meetup where I demonstrated Convoy, you can find it here. Thanks so much, and if you run into any issues, please visit our forums, and we’ll do our best to help you.