Using Containers and Elasticsearch Cluster for Twitter Monitoring

In the previous part of this series, we have seen how to deploy an Elasticsearch Cluster using Rancher Catalog. Now it’s time to make good use of this catalog, right?

Introduction

As a reminder, Elasticsearch is the cornerstone of the ELK platform (ELK stands for Elasticsearch/Logstash/Kibana). In this article, we’ll deploy the stack using Rancher Catalog, and use it to track tags and brands on Twitter. Tracking hashtags on Twitter can be very useful for measuring the impact of a Twitter-based marketing campaign. You can pull information like the number of times your announcement has been retweeted, or how many new followers your marketing campaign has brought in.

Installing the ELK Stack

Elasticsearch Following the previous article, you should know have a fully working Elasticsearch Cluster. So now for our example, we just have to tweak its configuration a bit, by creating an index template using JSON configuration.

• grab the JSON template here, on Github
• open your browser at http://[your rancher host where kopf is deployed]
• in kopf, click : ‘[more]’ and them ‘[index templates]’

Now we’ll give a name to our index template, and push its configuration.

• use twitter_elk_example as a template name
• paste the content of the JSON file you’ve previously downloaded
• click ‘[save]’

That’s it for our Elasticsearch Cluster. Now let’s move on to... Logstash Logstash allows you to process and transport these data right into your Elasticsearch cluster with a little bit of magic. Lots of data sources are natively supported (Twitter APIs, collectd, Apache logs, etc). While processing your data, Logstash can help you extract/format the right portion of your data. In doing so, you won’t push unnecessary or (even worse) wrong data, which would makes your Kibana dashboard irrelevant ! Before we start , we need to create Twitter Application Keys We’ll need specifically:

• Consumer Key
• Consumer Secret
• Access Token
• Access Token Secret

Note: make sure your Rancher hosts have their clocks synced, or you’ll won’t be able to use Twitter credentials correctly

Now go to the catalog page and select Logstash (latest version will do well). In the input field, you should add the following (replace the CAP text with your own twitter APIs authentication keys):

twitter {
 consumer_key => "INSERT YOUR CONSUMER KEY"
 consumer_secret => "INSERT YOUR CONSUMER SECRET"
 oauth_token => "INSERT YOUR ACCESS TOKEN"
 oauth_token_secret => "INSERT YOUR ACCESS TOKEN SECRET"
 keywords => [ "docker", "rancher_labs", "rancher", "kubernetes" ]
 full_tweet => true
 }

Note: in the keywords array, don’t use @ or #, or else Logstash will fail with an incorrect error: unauthorized message.

In the output field, you should then add:

output {
 elasticsearch {
 host => "elasticsearch:9200"
 protocol => "http"
 cluster_name => "NAME OF YOUR ELASTICSEARCH CLUSTER"
 index => "twitter_elk_example"
 document_type => "tweets"
 }

Finally, select : elasticsearch-clients as the Elasticsearch stack/service That’s it! Click on : [‘launch’] Rancher will do the magic and deploy Logstash, fully configured. If everything went well, within a few minutes you should see data being added to your Elasticsearch index. You can c[heck at http://[your ElasticSearch host]/#kopt] Kibana Kibana allows you to create a powerful dashboard using data stored in your Elasticsearch cluster. To deploy Kibana, you need to do just two things: select the correct version in Rancher Catalog (4.1.X), and connect it to the elasticsearch-clients container: That’s it! Kibana is ready to use, we’ll see configurations right afterwards. Now, the entire ELK stack is freshly deployed. While Elasticsearch and Logstash are already configured, we’ll need to take care of Kibana. In our case, we just need to import a JSON dashboard right into Kibana.

• grab the JSON file
• go to Settings --> Object, then click ‘I[mport]‘, and select the file previously downloaded. You should see something close to this:

All that’s left is to create the appropriate index settings in Kibana. Go to Indices , then click ‘[New]‘. You should be see the index created and @timestamp selected, as well. Here we are! You now have a Kibana dashboard that helps you monitor hashtags and brands on Twitter. You can load the imported dashboard clicking on its name here: After a few minutes, let’s check the dashboard again. You should then having something like this: Thanks for reading this tutorial on deploying a containerized ELK stack with Rancher! Please reach out to us on Github or on Twitter - we’d love to hear more about how you’re using Rancher in your organization. Rachid is a former virtualization consultant and Instructor. After a successful experience building and training the ops team of the French registry AFNIC, he is now the CIO of a worldwide recognized CRM and ecommerce agency.