There is a Docker Playground which is widely used to test the docker commands. In the same way, I would like to use the Kibana charts for multiple testing purposes, And also wanted to test the data loading/extracting the metric data in the elastic(formerly it was called elastic-search) The greatest capability of this stack is based on the 'Lucien search engine', which is the base for the Google search engine. The only path that you need to identify here is, how you can utilize the super-fast search capabilities and powerful visualizations with Kibana. In this post, I would like to explore the possible options to establish the playground for elastic-Kibana services, which will be integrated with the most widely used in the CNCF environments that is fluentd for data. we can also use the other option as an ELK playground learning lab. In that case, we need to use filebeat instead of fluentd. Note that earlier it was named logstash-forwarder now it is changed to filebeat.
 |
| Elastic-Kibana playground |
Prerequisites to build Elastic-Kibana playground
- Git/Git-bash installed
- Docker Engine (CE/EE) installed
- Docker-compose tool installed
- RAM: 8GB (minimum)
- Storage 40 GB
- Support of Virtual NIC
- CPU Cores: 2 cores with 2+GHz 64BIT Processors
Why elastic search?
In crisply we could make bullet points as:- Distributed
- real-time datastore
- real-time analysis
- Full-text search engine
- CNCF member and opensource
- Supports the Highly scalable elastic cluster
How elasticsearch is going to work?
The elastic-search will work with three important aspects - Indices, Shards, and Replicas. Let me tell you how they are interconnected to each other- An index is stored on a node, which is a part of a cluster Indices are broken down into shards. Each shard is either a primary or a replica. When expanding this deeper each log item is a document that contains fields and values.Configuration of elasticsearch instance
Basically, elasticsearch configuration is based on the three files.
- elasticsearch.yaml file where you can specify the requirements of each elasticsearch instance configurations, if clustered for HA then those details also defined.
- jvm.options for each elasticsearch instance.
- log4j2.properties for configuring elasticsearch logging details
How Elastic-Kibana playground works?
The recipe to built this Elastic-Kibana playground is using the docker-compose files which are available in the elastic tutorials. The YAML file uses the basic needs to run the elastic as a docker service that uses the current version of elastic docker images which are published here. This experiment I've used docker.elastic.co/elasticsearch/elasticsearch:7.5.1 in the es/Dockerfile. On the other hand Kibana latest image also used in kibana/Dockerfile to build with docker.elastic.co/kibana/kibana:7.5.1Getting Started
Here we have two options to run this elastic-kibana playground. One is on the Docker provided playground which will be available for 4 hours for testing purposes. Another one is you could do the same steps in your VM where all the prerequisites are satisfied. In your VM, If in case git is not installed then run this command `yum -y install git`.All Prerequisites are present then we are good to go! We have two choices: using the docker playground lab or you can use a Linux VM.
Login to your docker playground docker hub credentials
https://labs.play-with-docker.com/
Click on the 'Add NEW INSTANCE' in the left pane to get the SSH connection to the docker lab sandbox environment.
git --version docker -v docker-compose -v
Enter the following line in /etc/sysctl.conf file with the vm.max_map_count value as 262144, this is production setup. This is used to get this effected permanently
vi /etc/sysctl.conf vm.max_map_count=262144 # For temporary fix is on your playground you can run sysctl -w vm.max_map_count=262144
 |
| Pre-checks for playing in the elastic-kibana playground |
This will be having the git client already installed instance.
git clone https://github.com/BhavaniShekhar/elastic-kibana-playground.git cd elastic-kibana-playground ls -l
How to create and play in your elastic Kibana playground?
Using the latest version of docker-engine, run the following docker-compose command
docker-compose up -d
Check the Validation of the docker services list with the following command.
$ docker-compose images Container Repository Tag Image Id Size ------------------------------------------------------------------------------- es elastic-kibana-playground_es latest abc35cd273f0 760 MB fluentd elastic-kibana-playground_fluentd latest 16297b62ab90 47.4 MB kibana elastic-kibana-playground_kibana latest ea32b58ae8d0 905 MB $ docker-compose ps Name Command State Ports ------------------------------------------------------------------------------------- es /usr/local/bin/docker-entr ... Up 0.0.0.0:9200->9200/tcp, 9300/tcp fluentd tini -- /bin/entrypoint.sh ... Up 0.0.0.0:24224->24224/tcp, 5140/tcp kibana /usr/local/bin/dumb-init - ... Up 0.0.0.0:5601->5601/tcp
This docker-compose command will pull the three images and make them Up and running.
- elasticsearch
- fluentd
- Kibana
$ curl http://ip172-18-0-9-bockumjjagq000cgnmk0-9200.direct.labs.play-with-docker.com/
{
"name" : "es",
"cluster_name" : "dev-kibana-cluster",
"cluster_uuid" : "P_WeXeqzQnikD95IkUE5xQ",
"version" : {
"number" : "7.5.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
"build_date" : "2019-12-16T22:57:37.835892Z",
"build_snapshot" : false,
"lucene_version" : "8.3.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Know about Fluentd
The fluentd provides the log aggregation layer that allows you to filter, buffer, and route your logs from various applcations and platforms to a backend of your choosing. The fluentd community likes its performant, reliable, and flexible. Fluentd is marginally more performant than Logstash. Only noticeable if you're looking at logs in near real time. Either solution is a win.
Ready to play in Kibana!
You will be getting the 4 ports opened on your docker playground.
Play begins by using the Kibana port link
Troubleshoot points
You may encounter the issue at the beginning when the containers created and running the service but that may take longer time due to network latency. So you can check the corresponding container logs
You could see the logs of docker-compose command as well.
- Kibana will be using 5601
- fluentd will use 25224
- elasticsearch uses 9200 and 9300
 |
| playground OPEN PORT for Kibana, Elasticsearch fluentd ports |
 |
| Kibana playground loaded by using open port 5601 |
Troubleshoot points
You may encounter the issue at the beginning when the containers created and running the service but that may take longer time due to network latency. So you can check the corresponding container logs
You could see the logs of docker-compose command as well.
