Elasticsearch Analytics Tutorial

Table of Contents

• Introduction
• Connecting to Elasticsearch
• Querying With Search-Based Analytics
• Adding More Visualizations
• Summary

Introduction

Elasticsearch is a scalable full-text search engine with an HTTP web interface and schema-based JSON documents. Elasticsearch shines brightest when it is used in the background as the fundamental engine powering applications with convoluted search features and many requirements.

At it stands, Kibana’s position in Elastic’s popular ELK stack makes it the most common tool used for the purpose of visualizing and analyzing data from Elasticsearch. While Kibana certainly stands on its own merit, it also features some shortcomings; namely, it does not support integration with any other data source, nor does it offer user management features or support for raising proactive alerts.

At some point, Elasticsearch users may decide that one or more of these shortcomings are a deal breaker, and opt for a different visualization platform that addresses these shortcomings. That is where Knowi comes in. Knowi offers broad native integration to 35 other data sources in addition to Elasticsearch, plenty of user management features, and support for alerts, as well as search-based analytics and machine learning. If you’re interested in learning about how to use Knowi to visualize data from Elasticsearch, you’ve come to the right place.

Connecting to Elasticsearch

Once you’re logged in to your free Knowi trial account, you’ll need to set up your Elasticsearch datasource by following these steps: 

1. Head to the upper middle of the panel on the left side of your screen and click on “Data sources.”
2. Head to the top right section of the NoSQL Datasources that are listed and click on Elasticsearch.
3. Click on “Test Connection” at the bottom of your screen.
4. Once you’ve ensured that your connection is successful, click on “Save.”

Querying With Search-Based Analytics

Now that we’ve set up a datasource, it’s time to run a query on our data. Our goal is to analyze transit data from Georgia, and to specifically focus only on data which pertains to westbound vehicles on New Year’s Day a certain data set. Follow these steps in order to do so:

1. When you saved your datasource, you should’ve received an alert at the top of your screen saying “Datasource Added. Configure Queries.” Get started by clicking on “Queries.” Once you’ve done this, you will be taken to a query builder and you will see a new alert at the top of your screen which reads “Getting Indexes.” This means that Knowi is automatically indexing tables from the Elasticsearch datasource that you just connected to. Before you do anything else, name your query “Westbound Transit on New Year’s Day” under “Report Name*.”
2. Hover down to “Indexes.” Click inside the bar, and you will see every index that exists in your Elasticsearch Database. Click on “transit.” This will prompt Knowi’s broad native integration to generate an Elasticsearch JSON Query which calls the first 10,000 rows from all columns of the transit table.
3. Select “Preview” at the bottom left corner of your screen. This will show you preview data, but if you look at the visualization, you’ll notice Knowi automatically chose a map view. Because our preview data contains longitude and latitude coordinates, Know is smart enough to automatically turn the preview visualization into a Geo-Clusters/Custom Map visualization. If you click on one of the dots in the visualization, it will expand the map. This is exactly the type of visualization that we want to see, but remember, we’re only interested in westbound activity from New Year’s Day.
4. In order to filter our data, we’re going to use Knowi’s search-based analytics feature which is simple, intuitive, and appeals to non-technical users by allowing them to ask questions in plain English and receive results in real time. Simply type “show me westbound on 2017-01-01” and give it a second.
5. If you look back to your visualization, you should see that there are fewer observations on your map as you filtered it down to a specific subset of it. This is your cue that you’ve done everything right, and that now you just need to click “Save & Run Now” to complete your query.

Congratulations on setting up your first Elasticsearch query and visualization with Knowi!

Adding More Visualizations

As soon as you saved and ran your query, Knowi saved the raw data behind it as a dataset within Knowi’s elastic data warehouse, and it also saved your preview visualization as a widget within your Knowi account. In order to further analyze and visualize your data, follow these steps:

1. Widgets are built to exist on dashboards. Right now, you’ve got a widget without a home. Give your widget a home by maneuvering over to the left side panel and clicking “Dashboards.” Then, click on the “+” icon to create a new dashboard and name it “Westbound Transit Dashboard.” Click “OK” to save this dashboard; you will be immediately taken to it once you do.
2. Head back the panel on the left side of your screen, and just below “Dashboards,” click on “Widgets.” Here, you will see the “Westbound Transit on New Year’s Day” Widget which you just made. Add it to your dashboard by simply dragging it over and letting go.
3. Now that your widget is at home on your dashboard, you can conduct more analysis and visualization on your data. Click the 3 dot icon at the top right corner of your widget and select “Analyze” in order to be directed to the raw dataset which is currently empowering your widget.
4. Our goal here is to add visualize the distribution of different routes within our data. In other words, we want to see which routes were most frequently being traversed by westbound vehicles on New Year’s Day. So, from the left pane where the column names are listed, we’re going to drag “route_name” (make sure you choose this and not “route”) over to “Grouping/Dimensions.” Then, even though it already exists in “Fields/Metrics,” we’re going to drag it over there once more, but this time, we’re going to change the “Operation” in this second “route_name” field to “Count.” If you do this correctly, you will see six route names and the frequency with which they appear.
5. Drag the “Count of route_name” field from “Fields/Metrics” over to “Sort By” and then make sure to sort this field by descending order.
6. Now, head to the top of your screen, and next to “Data,” click on “Visualization.” Then, change “Visualization Type” to “Pie.”
7. Because we want to view our new pie chart side by side with our geo cluster map, we’re not going to save it; we’re going to clone it. To do this, head to the top right corner of your screen, find the “Clone” icon which looks like one piece of paper being dropped on top of another, and click on it. Then, name your new widget “Westbound Transit on New Year’s Day – Route Frequency.” Click “Clone” to complete this process and then click “Add to Dashboard” to add your new widget to your dashboard.

As you can see, Glenwood Road was our most traversed, with Moreland/Candler Park not far behind. There was a sizable drop between Moreland/Cander Park and Perry Blvd/West Highlands, and then a massive drop between Perry Blvd/West Highlands and the next three routes.

Summary

To summarize, we began this tutorial by establishing a successful connection to an Elasticsearch database and querying a specific subset of our transit dataset within our database using a helping hand from Knowi’s search-based analytics feature to make querying easier. When we successfully ran this query, Knowi stored the results of our query as a dataset in Knowi’s elastic data warehouse, and stored the preview visualization which we used as a widget in our Knowi account. We then created a new dashboard to serve as a home for our new widget, and created another widget to further analyze our dataset and answer an important question.