Elasticsearch Cluster Deployment using Rancher Catalog

Elasticsearch is a Lucene-based search engine developed by the open-source vendor, elastic. With principal features like scalability, resiliency, and top-notch performance, it has overtaken Apache Solr, one of its closest competitors. Nowadays, Elasticsearch is almost everywhere where a search engine is involved: it’s the E of the well-known ELK stack, which makes it straightforward for your project to process analytics (the L stands for Logstash which is used to process data like logs, streams, metrics; K stands for Kibana, a data visualization platform – projects also managed by elastic). Installing Elasticsearch from the Rancher Catalog Before we get started, let me tell you a bit about the Rancher catalog. The Rancher catalog uses rancher-compose and docker-compose to ease the installation of whatever tool you need. Using the Rancher catalog, you can deploy everything from a simple app like ghost (blogging platform) to a full CI/CD stack like GoCD. I’ll assume here that you have a fully working Rancher platform (a server and several nodes). If not, then head over to the Rancher documentation here, before going any further in this article and set up your environment. My environment looks like this (Figure 1, built using docker-machine on my laptop):

Figure *1: Elasticsearch Environment***

Accessing the Rancher catalog is simple:

• On the top menu bar of your Rancher UI, click on Catalog, then All.
• Using the search box on the upper right, search for *Elasticsearch***.
• You’ll see two versions of Elasticsearch are available (Figure 2). Both work fine, but for this article, we’ll stick to version on the left.
• Click on View Details. You will need to fill in some simple information (Figure 3).
• To fire up the installation, click Launch.

Figure *2: Elasticsearch Options in the Rancher Catalog***

Figure *3: Elasticsearch Data Form***

You should now see something similar to the image below (Figure 4). You can find more details about what Rancher is doing by clicking on the name of your stack (in my case, I’ve installed Elasticsearch, and named my stack LocalEs).

Figure *4: LocalEs app Naming Convention***

Expanding our view of the stack (Figure 5), we can see that deploying an Elasticsearch cluster using the Rancher catalog template has included:

• a Master node
• a Data node
• a Client node
• kopf, an Elasticsearch management web app

Figure *5: Elasticsearch Cluster Stack View***

Each of these nodes (except for kopf) comes with sidekick containers, which in this case are configuration and data volume containers. Your Elasticsearch cluster will be fully functional when all the entries are “active”. If you want to see how they are all connected to each other, take a look at the graph view (available from the drop-down menu on the right hand corner in Figure 6).

Figure *6: Elasticsearch Cluster Graph View***

Now, we can visualize how all these containers as they are mapped within the Rancher platform (Figure 7).

Figure *7: Elasticsearch Visual Map***

That’s it, our Elasticsearch cluster is now up and running. Let’s see how our cluster behaves! Cluster Management Depending on your Rancher setup, kopf is deployed on one of your Rancher nodes. You can access the application using http://[your kopf rancher host]. Here’s an example (Figure 8):

Figure *8: kopf node identification***

As you can see, everything seems to be fine, as kopf shows a green top bar. Indeed, our cluster is running without any data stored, so there’s no need for resiliency at this point. Let’s see how it goes if we manually create an index called ‘ranchercatalog’, with three shards and two replicas. Using curl, your query would be something like this:

curl -XPUT 'http://[your kopf rancher host]/es/ranchercatalog/' -d '{
   "settings" : {
      "index" : {
      "number_of_shards" : 3,
      "number_of_replicas" : 2
      }
   }
}'

Elasticsearch should reply {\“acknowledged\“:true}% Shards are related to data storage, and replicas to resiliency. This means our index will have its data stored using three shards but needs two more replicas of these shards. Now that our index has been successfully created, let’s talk a look at kopf.

Figure *9: kopf Status View***

As you can see in Figure 9, the top bar is now yellow, which indicates there may be something wrong with our Elasticsearch cluster. We can also see in the middle left of the page a warning sign (red triangle in Fig. 9) saying \“six unassigned shards.\” Remember when we created the ranchercatalog index, we specified:

• Three shards
• Two replicas

By default, the Elasticsearch Rancher catalog item deploys only 1 data node, so we need two more data nodes. Adding nodes can be easily done using the Rancher scale option. The results are shown in Figure 10.

Figure *10: Adding Nodes using Rancher Scale Option***

To scale your data nodes, let’s go again to Applications, then to Stack. Click on your stack, and then on elasticsearch-datanodes. You should have something like what is shown in Figure 10. Click 2 times on the + of the scale option and let Rancher do the work. You should see data nodes popping up one after another until you finally have something like what you see in Figure 11.

Figure *11: Node View to Verify Additions***

Let’s check if this is enough to bring back the beautiful green bar to kopf. Figure 12 provides the proof.

Figure *12: Corrected Nodes Verification***

Voilà! We now have a perfect and fully functional Elasticsearch Cluster. In my next post, we’ll see how to populate this index and do some cool queries! *Rachid is a former virtualization consultant and Instructor. * After a successful experience building and training the ops team of the French registry AFNIC, he is now the CIO of a worldwide recognized CRM and ecommerce agency.