Installing Rancher - From Single Container to High Availability

Update: This tutorial was updated for Rancher 2.x in 2019 here

Any time an organization, team or developer adopts a new platform, there are certain challenges during the setup and configuration process. Often installations have to be restarted from scratch and workloads are lost. This leaves adopters apprehensive about moving forward with new technologies. The cost, risk and effort are too great in the business of today. With Rancher, we’ve established a clear container installation and upgrade path so no work is thrown away. Facilitating a smooth upgrade path is key to mitigating against risk and increasing costs. This guide has two goals:

1. Take you through the installation and upgrade process from a technical perspective.
2. Inform you of the different types of installations and their purpose.

With that in mind, we’re going to walk through the set-up of Rancher Server in each of the following scenarios, with each step upgrading from the previous one:

• Single Container (non-HA) - installation
• Single Container (non-HA)- Bind mounted MySQL volume
• Single Container (non-HA) - External database
• Full Active/Active HA - (upgrading to this from our previous set up)

A working knowledge of Docker is assumed. For this guide, you’ll need one or two Linux virtual machines with the Docker engine installed and an available MySQL database server. All virtual machines need to be able to talk to each other, so be mindful of any restrictions you have in a cloud environment (AWS, GCP, Digital Ocean etc.). Detailed documentation is located here.

**Single Container (non-HA) – Installation

**

1. SSH into your Linux virtual machine
2. Verify your Docker installation with docker -v. You should see something resembling Docker version 1.12.x
3. Run sudo docker run -d --restart=unless-stopped -p 8080:8080 rancher/server
4. Docker will pull the rancher/server container image and run it on port 8080
5. Run docker ps -a. You should see an output similar to this: (Note: remember the name or ID of the rancher/server container)
6. At this point, you should be able to go to http://<server_ip>:8080 in your browser and see the Rancher UI.
   

You should see the Rancher UI with the welcome modal: Since this is our initial set up, we need to add a host to our Rancher environment:

1. Click ‘Got it!’
2. Then click ‘Add a Host’. This first time you’ll see a Host Registration URL page:
3. For this article, we’ll just go with whatever IP address we have. Click ‘Save’.
4. Now, click ‘Add a Host’ again. You’ll see this: (note: the ports that have to been open for hosts to be able to communicate are 500 and 4500.) From here you can decide how you want to add your hosts based on your infrastructure.
5. After adding your host(s) you should see something like this:

So, what’s going on here? The rancher-agent-bootstrap container runs once to get the rancher-agent up and running then stops (notice the red circle indicating a stopped container). As we can see above, the health check container is starting up. Once all infrastructure containers are up and running on the host you’ll see this:

Here we see all infrastructure containers (health check, scheduler, metadata, network manager, IPsec, and cni-driver) are all up and running on the host.

Tip: to view only user containers, uncheck ‘Show System’ in the top right corner of the Host view. Congratulations! You’ve set up a Rancher Server in a single container. Rancher is up and running and has a local MySQL database running inside of the container. You can add items from the catalog, deploy your own containers etc. As long as you don’t delete the rancher/server container, any changes you make to the environment will be preserved as we go to our next step.

**Single Container (non-HA) – Bind-mounted volume

**

Now we’re going to take our existing Rancher server and upgrade it to use a bind-mounted volume for the database. This way, should the container die when we upgrade to a later version of Rancher, we don’t lose the data for what we’ve built. In our next steps, we’re going to stop the rancher-server container, externalize the data to the host, then start a new instance of the container using the bind-mounted volume. Detailed documentation is located here.

1. Let’s say our rancher/server container is named fantastic_turtle.
2. Run docker stop fantastic_turtle.
3. Run docker cp fantastic_turtle:/var/lib/mysql <path on host> (Any path will do but using /opt/docker or something similar is not recommended). I use /data as it’s usually empty. This will copy the database files out of the container to the file system to /data. The export will put your database files at /data/mysql.
4. Verify the location by running ls -al /data You will see a mysql directory within the path.
5. Run sudo chown -R 102:105 /data. This will allow the mysql user within the container to access the files.
6. Run docker run -d -v /data/mysql:/var/lib/mysql -p 8080:8080
   --restart=unless-stopped rancher/server:stable. Give it about 60 seconds to start up.
7. Open the Rancher UI at http://<server_ip>:8080. You should see the UI exactly as you left it. You’ll also notice your workloads that you were running have continued to run.
8. Let’s clean up the environment a bit. Run docker ps -a.
9. You’ll see 2 rancher/server Image containers. One will have a status of Exited (0) X minutes ago and one will have a status of Up X minutes. Copy the name of the container with exited status.
10. Run docker rm fantastic_turtle.
11. Now our docker environment is clean with Rancher server running with the new container.

**Single Container (non-HA) - External database

**

As we head toward an HA set up, we need have Rancher server running with an external database. Currently, if anything happens to our host, we could lose the data supporting the Rancher workloads. We’re going to launch our Rancher server with an external database. We don’t want to disturb our current set up or workloads so we’ll have to export our data, import into a proper MySQL or MySQL compliant database and restart our Rancher server that points to our external database with our data in it.

1. SSH into our Rancher server host.
2. Run docker exec -it <container name> bash. This will give you a terminal session in your rancher/server container.
3. Run mysql -u root -p.
4. When prompted for a password, press [Enter].
5. You now have a mysql prompt. 6.Run show databases. You’ll see this: This way we know we have the rancher/server database.
6. Run exit.
7. Run mysqldump -u root -p cattle > /var/lib/mysql/rancher-backup.sql When prompted for a password hit [Enter].
8. Exit the container. 10.Run ls -al /data/mysql. You’ll see your rancher-backup.sql in the directory. We’ve exported the database! At this point, we can move the data to any MySQL compliant database running in our infrastructure as long as our rancher/server host can reach the MySQL database host. Also, keep in mind all this while your workloads that you have been running on the Rancher server and hosts are fine. Feel free to use them. We haven’t stopped the server yet, so of course they’re fine.
9. Move your rancher-backup.sqlto a target host running a MySQL database server.
10. Open a mysql session with your MySQL database server. Run mysql -u <user> -p.
11. Enter your decided or provided password.
12. 14. Run CREATE DATABASE IF NOT EXISTS cattle COLLATE = ‘utf8_general_ci’ CHARACTER SET = ‘utf8’;
13. Run GRANT ALL ON cattle.* TO ‘cattle’@‘%’ IDENTIFIED BY ‘cattle’; This creates our cattle user for the cattle database using the cattle password. (note: use a strong password for production)
14. Run GRANT ALL ON cattle.* TO ‘cattle’@‘localhost’ IDENTIFIED BY ‘cattle’; This will allow us to run queries from the MySQL database host.
15. Find where you put your rancher-backup.sql file on the MySQL database host. From there, run mysql -u cattle -p cattle < rancher-backup.sql This says \“hey mysql, using the cattle user import this file into the cattle database\“. You can also use root if you prefer.
16. Let’s verify the import. Run mysql -u cattle -p to get a mysql session.
17. Once in, run use cattle; Then show tables; You should see something like this:

Now we’re ready to bring up our Rancher server talking to our external

database.

1. Log into the host where Rancher server is running.
2. Run docker ps -a. Again, we see our rancher/server container is running:
3. Let’s stop our rancher/server. Again, our workloads will continue to run. Run docker stop <container name>’.
4. Now let’s bring it up using our external database. Run docker run -d --restart=unless-stopped -p 8080:8080 rancher/server --db-host <mysql host> --db-port 3306 --db-user cattle --db-pass cattle --db-name cattle. Give it about 60+ seconds for the rancher/server container to run.
5. Now open the Rancher UI at http://<server_ip>:8080.

Congrats! You’re now running Rancher server with an external database and your workloads are preserved.

**Rancher Server - Full Active/Active HA

**

Now it’s time to configure our Rancher server for High Availability. Running Rancher server in High Availability (HA) is as easy as running Rancher server using an external database, exposing an additional port, and adding in an additional argument to the command so that the servers can find each other. 1. Be sure that port 9345 is open between the Rancher server host and any other hosts we want to add to the cluster. Also, be sure port 3306 is open between any Rancher server and the MySQL server host. 2. Run docker stop <container name>. 3. Run docker run -d --restart=unless-stopped -p 8080:8080 -p 9345:9345 rancher/server
--db-host <mysql host> --db-port 3306 --db-user cattle --db-pass cattle --db-name cattle \ --advertise-address <IP_of_the_Node> (*note: Cloud provider users should use the internal/private IP address). Give it 60+ seconds for the container to run. (note: if after 75 seconds you can’t view the Rancher UI, see the troubleshooting section below) 4. Open the Rancher UI at http://<server_ip>:8080. You’ll see all your workloads and settings as you left them. 5. Click on Admin then High Availability. You should see your single host you’ve added. Let’s add another node to the cluster. 6. On another host, run the same command but replacing the --advertise-address <IP_of_the_Node> with the IP address of the new host you’re adding to the cluster. Give it 60+ seconds. Refresh your Rancher server UI. 7. Click on Admin then High Availability. You should see both nodes have been added to your cluster. 8. Because we recommend an odd number of Rancher server nodes, add either 1 or 3 more nodes to the cluster using the same method. Congrats! You have a Rancher server cluster configured for High Availability.

Troubleshooting & Tips

During my time walking through these steps myself I ran into a few issues. Below are some you might run into and how to deal with them. Issue: Can’t view the Rancher UI after 75 seconds. 1. SSH into the Rancher server host. 2. Confirm rancher/server is running. Run docker ps –a. Given an output like this: 3. To view logs, run `docker logs –t tender_bassi` (in this case). If you see something like this: It’s Rancher being unable to reach the database server or authenticate with the credentials we’ve provided it in our start up command. Take a look at networking settings, username and password and access privileges in the MySQL server.

Tip: While you may be tempted to name your rancher/server ‘—name=rancher-server’ or something like it this is not recommended. The reason for this is if you need to rollback to your prior container version after an upgrade step, you’ll have clear distinction between container versions.

Conclusion

So, what have we done? We’ve installed Rancher server as a single container. We’ve upgraded the Rancher installation to a high availability platform instance without impacting running workloads. We’ve also established guidelines for different types of environments. We hope this was helpful. Further details on upgrading are available here https://rancher.com/docs/rancher/v1.6/en/upgrading/.