{"id":2425,"date":"2015-07-24T11:18:34","date_gmt":"2015-07-24T09:18:34","guid":{"rendered":"https:\/\/test.viaboxx.de\/2015\/07\/24\/easily-generate-live-heatmaps-for-geolocations-with-elk\/"},"modified":"2021-08-10T21:38:55","modified_gmt":"2021-08-10T21:38:55","slug":"easily-generate-live-heatmaps-for-geolocations-with-elk","status":"publish","type":"post","link":"https:\/\/www.viaboxx.de\/en\/blog\/easily-generate-live-heatmaps-for-geolocations-with-elk\/","title":{"rendered":"Easily generate live heatmaps for geolocations with ELK"},"content":{"rendered":"

Here at Viaboxx we are using the ELK Stack (Elasticsearch<\/a>, Logstash<\/a> and Kibana<\/a>) to have a centralised but scalable logging infrastructure. In this blog you will learn how to use this stack to easily have a heat map for geo coordinates.<\/p>\n

Context<\/strong><\/p>\n

In our software we are using \u00a0the Logstash-Logback encoder\u00a0<\/a>\u00a0to directly have JSON\u00a0logs ready for logstash. Those are either directly send to Logstash (behind a Redis queue) or saved to a file and send via the Logstash-Forwarder, depending on the infrastructure and firewalls.<\/p>\n

Those logs are then preprocessed in Logstash and stored in the Elasticsearch document store. We are using the new Kibana 4 web interface to allow searching through the logs of all hosts or visualise log metrics.<\/p>\n

One of our products is a route optimizer<\/strong>, which uses\u00a0geodata for tour optimization. Kibana 4 provides a heatmap visualization right out of the box, one just needs to have the data prepared for it to work.<\/p>\n

If you don’t have the geodata already in Elasticsearch, don’t hesitate!\u00a0If you have the ELK stack and you have a CSV files of geolocations you can generate your heatmap in a few lines of code.<\/p>\n

Tutorial<\/strong><\/p>\n

Let’s say you have a simple CSV ‘test.csv’ with just latitudes and longitudes, the heatmap should indicate how often a geolocation was found in the data (within a distance).<\/p>\n

[codesyntax lang=\"text\"]<\/pre>\n
#Latitude, Longitude\n50.98474812, 7.47645034\n50.98474812, 7.47645034\n50.98474812, 7.47645034\n50.62703234,7.74493172\n50.96807514, 6.99762216\n50.78432438,7.64722746\n51.13014686,7.01966161\n50.55503241,6.56177651\n50.68262437,7.5592885\n50.59502431,6.46768414\n50.59502431,6.46768414\n50.55442372,7.60962654\n50.66100656,6.45276664\n50.66100656,6.45276664\n51.05079311,6.77458391\n50.93658508,6.70941164\n50.50670651,6.69422478<\/pre>\n
[\/codesyntax]<\/pre>\n
We will process this data with\u00a0a small Logstash configuration, which stores the data into Elasticsearch. Kibana 4 does not directly visualise the latitude and longitude of a document, but requires a geohash instead (nicely described \u00a0in this documentation<\/a>). This geohash needs to be mapped in advance before storing data.<\/p>\n
At first, we will\u00a0create the index in Elasticsearch and define the geohash mapping. Let’s name the index “geostore<\/em>” and the document will be named “locality<\/em>” with a property named “location<\/em>“. We will assume the you are on the ELK host and are using the default ports (maybe you have it run locally via Docker).<\/p>\n
$ curl -XPUT 'http:\/\/localhost:9200\/geostore'<\/pre>\n
$ curl -XPUT 'http:\/\/localhost:9200\/geostore\/_mapping\/locality' -d '\n{\n   \"locality\" : {\n       \"properties\" : {\n              \"location\" : {\n                  \"type\" : \"geo_point\",\n                  \"geohash_prefix\":     true,\n                  \"geohash_precision\":  \"1km\"\n              }\n          }\n       }\n}'<\/pre>\n
This mapping of location defines that it is of type geo_point<\/em>, and it defines the precision of the geohash. If you want to have a finer or coarse comparison, change this value accordingly to your needs.<\/p>\n
If later you need to restart configuring the mapping, delete the index and start again. This is the easiest way in this example, don’t do this in production. This command deletes the index:<\/p>\n
$ curl -XDELETE 'http:\/\/localhost:9200\/geostore'\n<\/pre>\n
Now that we have the Elasticsearch index ready, we can configure Logstash to directly store the data of the “test.csv<\/em>” into the “geostore<\/em>” \u00a0index.<\/p>\n
This is the config (named “geostore.conf<\/em>“) required for \u00a0our Logstash setup:<\/p>\n
[codesyntax lang=\"text\"]<\/pre>\n
input {\n    stdin {}\n}\nfilter {\n  # Step 1, drop the csv header line\n  if [message] =~ \/^#\/ {\n    drop {}\n  }\n  \n  # Step 2, split latitude and longitude\n  csv {\n    separator => ','\n    columns => [ 'lat', 'lon' ]\n  }\n\n  # Step 3\n  # move lat and lon into location object \n  # for defined geo_point type in ES\n  mutate {  \n    rename => [ \"lat\", \"[location][lat]\", \"lon\", \"[location][lon]\" ]\n  }\n}\noutput {\n  elasticsearch {\n    host => 'localhost'\n    index => 'geostore'\n    document_type  => \"locality\"\n    flush_size => 1000\n    protocol => 'http'\n  }\n}<\/pre>\n
[\/codesyntax]<\/pre>\n
This will parse the csv (given as standard input), drop the header line and parse the latitude and longitude values.<\/p>\n
The mutate moves lat<\/em>\u00a0and lon<\/em>\u00a0into the location object that we have predefined to be a geo_type<\/em>. This is the most important line, as this is already the correct structure that Elasticsearch and Kibana 4 expect.<\/p>\n
On the host, with Logstash just execute:<\/p>\n
$ cat test.csv | \/opt\/logstash\/bin\/logstash -f geostore.conf<\/pre>\n
This will take a few seconds to startup logstash, parse the input and store the result into Elasticsearch.<\/p>\n
Now that we have the data in Elasticsearch, let’s move to Kibana 4. After logged into Kibana, you need to add the index to Kibana.<\/p>\n
Go to: Settings -> Indices -> Add New -> Write<\/em>\u00a0“geostore<\/em>” in the index name field.<\/p>\n
After you add the index, you’ll see all fields in \u00a0the documents of the index, especially you should check if the property location<\/em>\u00a0is \u00a0classified as geo_point.<\/p>\n
\"Kibana-IndicesSettings\"<\/a>
Properties identified in the defined geostore Index of Elasticsearch<\/figcaption><\/figure>\n
Now we are nearly done, go to Visualize -> Tile map -> From a new search -> Select “geostore”<\/em> index.<\/p>\n
The default value as metric is count, this counts how often a geohash is counted in the index with the given time interval.<\/p>\n
Select as bucket type “Geo coordinates<\/em>” and then already geohash<\/em>\u00a0should be selected as aggregation and location<\/em>\u00a0as field. The default visualisation for a Tile map is a circled view. As we want to have a heatmap, \u00a0select “options<\/em>” and change the map type to “heatmap<\/em>“. \u00a0After pressing “Apply \u00a0changes<\/em>” you will already see the map and can zoom in to \u00a0see the different locations as a heat map! Congratulations!<\/p>\n
\"kibana4-exampleheatmap\"<\/a>
Kibana 4 showing the geostore heatmap.<\/figcaption><\/figure>\n
If you don’t see the data, make sure the change the time interval that Kibana uses for its query (on the top right). The default is 15 minutes into the past. Depending on when you imported the data into Elasticsearch. This feature really shines if you have geo locations in your production data and for example want to see which areas where the focus of deliveries in the last week.<\/p>\n","protected":false},"excerpt":{"rendered":"
Here at Viaboxx we are using the ELK Stack (Elasticsearch, Logstash and Kibana) to have a centralised but scalable logging infrastructure. In this blog you will […]<\/p>\n","protected":false},"author":4,"featured_media":6952,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_uag_custom_page_level_css":"","site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[76],"tags":[84,85,86,87,88],"class_list":["post-2425","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-code","tag-elasticsearch","tag-geolocation","tag-heatmap","tag-kibana","tag-logstash"],"uagb_featured_image_src":{"full":["https:\/\/www.viaboxx.de\/wp-content\/uploads\/2015\/07\/Blog-Post-easily-generate-live-heatmaps-for-geolocations-with-elk.jpg",2200,700,false],"thumbnail":["https:\/\/www.viaboxx.de\/wp-content\/uploads\/2015\/07\/Blog-Post-easily-generate-live-heatmaps-for-geolocations-with-elk-150x150.jpg",150,150,true],"medium":["https:\/\/www.viaboxx.de\/wp-content\/uploads\/2015\/07\/Blog-Post-easily-generate-live-heatmaps-for-geolocations-with-elk-300x95.jpg",300,95,true],"medium_large":["https:\/\/www.viaboxx.de\/wp-content\/uploads\/2015\/07\/Blog-Post-easily-generate-live-heatmaps-for-geolocations-with-elk-768x244.jpg",768,244,true],"large":["https:\/\/www.viaboxx.de\/wp-content\/uploads\/2015\/07\/Blog-Post-easily-generate-live-heatmaps-for-geolocations-with-elk-1024x326.jpg",1024,326,true],"1536x1536":["https:\/\/www.viaboxx.de\/wp-content\/uploads\/2015\/07\/Blog-Post-easily-generate-live-heatmaps-for-geolocations-with-elk-1536x489.jpg",1536,489,true],"2048x2048":["https:\/\/www.viaboxx.de\/wp-content\/uploads\/2015\/07\/Blog-Post-easily-generate-live-heatmaps-for-geolocations-with-elk-2048x652.jpg",2048,652,true]},"uagb_author_info":{"display_name":"Jan Nonnen","author_link":"https:\/\/www.viaboxx.de\/en\/blog\/author\/jan-nonnenviaboxx-de\/"},"uagb_comment_info":2,"uagb_excerpt":"Here at Viaboxx we are using the ELK Stack (Elasticsearch, Logstash and Kibana) to have a centralised but scalable logging infrastructure. In this blog you will […]","_links":{"self":[{"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/posts\/2425"}],"collection":[{"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/comments?post=2425"}],"version-history":[{"count":6,"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/posts\/2425\/revisions"}],"predecessor-version":[{"id":6950,"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/posts\/2425\/revisions\/6950"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/media\/6952"}],"wp:attachment":[{"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/media?parent=2425"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/categories?post=2425"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.viaboxx.de\/en\/wp-json\/wp\/v2\/tags?post=2425"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}