Migrating a Rancher 2.1.x Single Node Installation to a High Availability Installation

Rancher offers two installation methods, single node and high availability. Single node installations allow users to quickly deploy a working Rancher 2 installation suitable for short-term development or proof-of-concept purposes. High availability deployments are significantly better suited for long-lived deployments of Rancher.

Important Notes

• Migration from Single Node to HA is not supported by Rancher Labs
• You should be familiar with the Rancher architecture/troubleshooting in the event that you have issues during this process

Prerequisites

In order to migrate a single node Rancher installation to a high availability installation, a few facts must be true.

• You should be running a Rancher version that is within 2.1.x and an RKE version within 0.1.x.
• The server-url parameter must be a DNS name that can be changed to point to the HA Rancher installation. If your server-url is an IP address, you must first change the server-url to a DNS name and update the node/cluster agents of your user clusters in order to allow the nginx ingress controller of the HA cluster to properly route Rancher traffic after migration. If this is not done, you will lose access to all of your workload clusters.
• You must be able to provision new instances that will be dedicated to running Rancher in HA. Performing an in-place migration is very dangerous and there is no easy rollback strategy. It is highly recommended to consult the recommended architecture.
• You should familiarize yourself with the architecture differences between a single-node installation and high-availability installation.
• Your CA should not be changed from your single node installation. This means that if you are using custom certificates, you should continue to use the custom certificates.

Migration Tutorial

The process of migrating from a single node to high availability installation of Rancher can be summarized in a few steps:

On Rancher Single Node Instance:

1. Back up the Rancher single node container
2. Assemble a pki.bundle.tar.gz using the certificates found within the single node container
3. Run a temporary utility container to perform an etcd snapshot of the embedded etcd running within the container
4. Stop the old Rancher single-node container
5. Change your DNS record for server-url to point to your new HA load balancer

On Your Workstation or Bastion Host

1. Transfer the generated pki.bundle.tar.gz and single-node-etcd-snapshot from your Rancher single-node instance to your workstation in a (preferably empty) working directory
2. Generate an rke cluster configuration file pointing to your new HA nodes
3. rke etcd snapshot-restore the snapshot you retrieved from the single-node container
4. rke up
5. Install Rancher in HA by following the documentation

Before You Begin

Throughout the tutorial, you’ll enter a series of commands, replacing placeholders with data from your environment. These placeholders are denoted with angled brackets and all capital letters (<EXAMPLE>).

A table of the various placeholders used throughout this tutorial can be found below. Write down or copy this information before you begin.

On Rancher Single Node Instance

Step 1. Back up the Rancher Single-Node Container

First, you should back up the Rancher single-node container to ensure that you can rollback to running Rancher single-node should you have problems during your migration. More information about this can be found here: https://rancher.com/docs/rancher/v2.x/en/backups/backups/single-node-backups/

Step 2. Assemble a pki.bundle.tar.gz using the certificates found within the single node container

First, docker exec into the Rancher container:

[root@single-node-rancher ~]# docker exec -it <RANCHER_CONTAINER_NAME> /bin/bash

Once inside the container, tar the /etc/kubernetes/ssl directory into a pki.bundle.tar.gz:

root@9f4b1729d8ca:/var/lib/rancher# tar -zcvf pki.bundle.tar.gz /etc/kubernetes/ssl
root@9f4b1729d8ca:/var/lib/rancher# exit

Transfer out the pki.bundle.tar.gz that was just generated to your current working directory on the single node instance:

[root@single-node-rancher ~]# docker cp <RANCHER_CONTAINER_NAME>:/var/lib/rancher/pki.bundle.tar.gz .

Step 3. Run a temporary utility container to perform an etcd snapshot of the embedded etcd running within the container

docker run a temporary rke-tools container called etcd-utility with your Rancher container network attached and mount the current working directory (which should have your pki.bundle.tar.gz into it:

docker run --net=container:<RANCHER_CONTAINER_NAME> -it -v $(pwd):/cwd --name etcd-utility rancher/rke-tools:v0.1.20

Create a folder called ssl within this container that you will extract the pki.bundle.tar.gz to:

bash-4.4# mkdir ssl && cd ssl
bash-4.4# cp /cwd/pki.bundle.tar.gz .
bash-4.4# tar -zxvf pki.bundle.tar.gz --strip-components 3

Snapshot your single-node etcd into a file called single-node-etcd-snapshot

bash-4.4# cd /
bash-4.4# ETCDCTL_API=3 etcdctl snapshot save --cacert=/ssl/kube-ca.pem --cert=/ssl/kube-etcd-127-0-0-1.pem --key=/ssl/kube-etcd-127-0-0-1-key.pem single-node-etcd-snapshot
bash-4.4# exit

Copy the etcd snapshot from the etcd-utility container to your current working directory

[root@single-node-rancher ~]# docker cp etcd-utility:/single-node-etcd-snapshot .

Your current working directory on your single-node instance should contain two files, pki.bundle.tar.gz and single-node-etcd-snapshot. These are the two components required to migrate Rancher from single-node to HA.

Step 4: Stop the old Rancher single node container

Stop the old rancher single-node container

[root@single-node-rancher ~]# docker stop <RANCHER_CONTAINER_NAME>

Step 5: Change your DNS record for server-url to point to your new HA load balancer

In order to properly migrate Rancher, you should update the DNS record in your DNS infrastructure to point the Rancher server-url to your new HA load balancer.

On Your Workstation or Bastion Host

Transfer the generated pki.bundle.tar.gz and single-node-etcd-snapshot from your Rancher single-node instance to your workstation in a (preferably empty) working directory

Endeavor:single-node-to-ha-migration chriskim$ scp root@<RANCHER_SINGLE_NODE_HOST>:/root/pki.bundle.tar.gz .
Endeavor:single-node-to-ha-migration chriskim$ scp root@<RANCHER_SINGLE_NODE_HOST>:/root/single-node-etcd-snapshot .

Generate an rke cluster configuration file pointing to your one of your new HA nodes

An example rancher-cluster.yml file looks like follows:

nodes:
  - address: <RANCHER_HA_HOST_1>
    user: centos
    role: [controlplane,worker,etcd]

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

Make an /opt/rke/etcd-snapshots directory on your desired HA node and copy the single-node-etcd-snapshot and pki.bundle.tar.gz files to that directory:

Endeavor:single-node-to-ha-migration chriskim$ ssh root@<RANCHER_HA_HOST_1> "mkdir -p /opt/rke/etcd-snapshots"
Endeavor:single-node-to-ha-migration chriskim$ scp pki.bundle.tar.gz root@<RANCHER_HA_HOST_1>:/opt/rke/etcd-snapshots
Endeavor:single-node-to-ha-migration chriskim$ scp single-node-etcd-snapshot root@<RANCHER_HA_HOST_1>:/opt/rke/etcd-snapshots

Restore the single-node-etcd-snapshot to your new HA node with RKE:

rke etcd snapshot-restore --name single-node-etcd-snapshot --config rancher-cluster.yml

INFO[0000] Starting restoring snapshot on etcd hosts
INFO[0000] [dialer] Setup tunnel for host [ha-rancher01.fmt01.rancher.com]
INFO[0000] [hosts] Cleaning up host [ha-rancher01.fmt01.rancher.com]
INFO[0000] [hosts] Running cleaner container on host [ha-rancher01.fmt01.rancher.com]
INFO[0000] [kube-cleaner] Pulling image [rancher/rke-tools:v0.1.15] on host [ha-rancher01.fmt01.rancher.com]
INFO[0004] [kube-cleaner] Successfully pulled image [rancher/rke-tools:v0.1.15] on host [ha-rancher01.fmt01.rancher.com]
INFO[0004] [kube-cleaner] Successfully started [kube-cleaner] container on host [ha-rancher01.fmt01.rancher.com]
INFO[0004] [hosts] Removing cleaner container on host [ha-rancher01.fmt01.rancher.com]
INFO[0004] [hosts] Removing dead container logs on host [ha-rancher01.fmt01.rancher.com]
INFO[0005] [cleanup] Successfully started [rke-log-cleaner] container on host [ha-rancher01.fmt01.rancher.com]
INFO[0005] [remove/rke-log-cleaner] Successfully removed container on host [ha-rancher01.fmt01.rancher.com]
INFO[0005] [hosts] Successfully cleaned up host [ha-rancher01.fmt01.rancher.com]
INFO[0005] [etcd] Restoring [single-node-etcd-snapshot] snapshot on etcd host [ha-rancher01.fmt01.rancher.com]
INFO[0005] [etcd] Pulling image [rancher/coreos-etcd:v3.2.18] on host [ha-rancher01.fmt01.rancher.com]
INFO[0007] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.18] on host [ha-rancher01.fmt01.rancher.com]
INFO[0007] [etcd] Successfully started [etcd-restore] container on host [ha-rancher01.fmt01.rancher.com]
INFO[0007] [etcd] Building up etcd plane..
INFO[0007] [etcd] Successfully started [etcd] container on host [ha-rancher01.fmt01.rancher.com]
INFO[0007] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [ha-rancher01.fmt01.rancher.com]
INFO[0007] [etcd] Successfully started [etcd-rolling-snapshots] container on host [ha-rancher01.fmt01.rancher.com]
INFO[0012] [certificates] Successfully started [rke-bundle-cert] container on host [ha-rancher01.fmt01.rancher.com]
INFO[0013] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [ha-rancher01.fmt01.rancher.com]
INFO[0013] [etcd] Successfully started [rke-log-linker] container on host [ha-rancher01.fmt01.rancher.com]
INFO[0013] [remove/rke-log-linker] Successfully removed container on host [ha-rancher01.fmt01.rancher.com]
INFO[0013] [etcd] Successfully started etcd plane..
INFO[0013] [certificates] Successfully started [rke-bundle-cert] container on host [ha-rancher01.fmt01.rancher.com]
INFO[0013] [certificates] successfully extracted certificate bundle on host [ha-rancher01.fmt01.rancher.com] to backup path [/etc/kubernetes/.tmp/]
INFO[0013] Finished restoring snapshot [single-node-etcd-snapshot] on all etcd hosts

At this point, you can run an rke up --config rancher-cluster.yml alongside the rest of the HA installation steps. It is highly recommended that you change your DNS record for rancher now, so that your user clusters are able to connect back to rancher once you install it following the HA instructions. You may need to run rke up --config rancher-cluster.yml twice to ensure that all addon jobs are successful.

Once you have successfully installed Kubernetes with your Rancher single node backup restored to your instance, you can add your additional HA instances to the rancher-cluster.yml and run rke up --config rancher-cluster.yml to add these nodes to your cluster.

At this point, you can continue installing Rancher using the documentation.

Cleaning Up

If your migration is successful, you can clean up by removing the Rancher container from your single node instance and/or removing the instance altogether.

[root@single-node-rancher ~]# docker rm <RANCHER_CONTAINER_NAME>

Rolling Back

In the event that your migration is unsuccessful, rollback can be achieved through the following two conditions:

• Change your DNS entry for server-url back to your single node Rancher instance
• Start the <RANCHER_CONTAINER_NAME> on your Rancher single node instance
  

[root@single-node-rancher ~]# docker start <RANCHER_CONTAINER_NAME>