Monitoring and Alert management tricks using Promtool and amtool

By -
These days I am busy and exploring the most critical stuff of the cloud that is Monitoring. There were many ways to monitor the 'Kubernetes Cluster', CNCF recommendation on 'metric' data is 'Prometheus' stack that is using Grafana, AlertManager.

Install Prometheus your laptop

I've chosen the Prometheus for Windows amd64 option as my system with Windows 7.
download link

Validating Prometheus alert rules

After creating the Alert rule how do I check it is correct syntax or not. There is a nice tool that comes with Prometheus installation. Let me demonstrate the scenario where I've selected 7 important alert rules for my project.

set PROMETHEUS_HOME=C:\Users\pavan\myprom\prometheus-2.8.1.windows-amd64
set AM_HOME=C:\Users\pavan\myprom\alertmanager-0.16.1.windows-amd64
set PATH=%PROMETHEUS_HOME%;%AM_HOME%;%PATH%

After Setting the PATH you can validate the alerting rules with the following command:
promtool check rules my-mon_alertrules.yml
Checking my-mon_alertrules.yml
  SUCCESS: 7 rules found

Here is my rules sample I've defined the 7 alerts and it validated them the count and checks the YAML syntax.

Check Prometheus Scraping from targets

Every target Container will be pushing the metrics through the cluster Port that exposed, then the URL will be having context path as /metrics. To check the metrics are active or not by using simple 'curl' command followed by the exposed IP:Port/metrics executed inside container. Same should work from the Prometheus container as well. You will see the outcome of the corresponding metrics if it is working, otherwise, connection refused error will be thrown.

curl localhost:10413/metrics

There could be possibility of misconfiguration either in the target Container or else Prometheus not in healthy condition.



Checking AlertManager configuration with amtool


To know about what all the configurations are defined in current AlertManager working use the following command :
 ./amtool --alertmanager.url http://localhost:9501/alertmanager config
Note that hostname, port will be changed according to your configurations. Check for the silence query
./amtool --alertmanager.url http://localhost:7501/alertmanager silence query

Comments

Post a Comment

Popular posts from this blog

Ansible Jinja2 Templates: A Complete Guide with Examples

By -
Image
Here I'm starting this post with famous ARISTOTLE quote. "We are what we repeatedly do. Excellence, then is not an act, but a Habit" Welcome back Ansible Automations and who are habituated as Ansible automation specialists this post for them to give boosted walk through, In this post, I would like to share my experiments with Jinja2 templates usage on the Ansible playbook. Of-course Jinja is from Japan it is world famous for templatization capabilities badvanced Ansible templating techniquesy integrating with multiple languages such as Python, Ruby, Salt talk etc. In this post we will be cover the following topics: What is Jinja2 template do in Ansible? Template with filters Template with Lists and sets Template module in Ansible Template with Flow Control Template using Looping Template inheritance** What is Jinja2 template do in Ansible? Jinja2 is another Python library created for Flask web framework, that comes part of Ansible installation. It is special ability as i...
Read more

Ansible 11 The uri module with examples

By -
Image
 Hey DevSecOps Automation specialist, Welcome back to the DevOps Hunter blog. In this post, I made it for learning more about all possible parameters that we can use when an application validation is done with the Ansible 'uri' module.  Why uri module? Web Application returns status HTTPCode 200 for success, 404 for failures, and also for 503 for Server internal issues. When you work on the restart of a web application we need to know the status of the application to proceed with the next move. So this uri module is most important for reboot and restart of web applications using ansible. The uri module parameters Supported parameters include: attributes, backup, body, body_format, client_cert, client_key, content, creates, delimiter, dest, directory_mode, follow, follow_redirects, force, force_basic_auth, group, headers, http_agent, method, mode, owner, regexp, remote_src, removes, return_content, selevel, serole, setype, seuser, src, status_code, timeout,...
Read more

Jenkins Active choices parameter - Dynamic input

By -
Image
Hello DevOps team!! Today I've revisited the experiment with the Jenkins ACTIVE CHOICE Parameter  to get the Dynamic parameters effect on the Build Job parameters. Installation Active Choice parameter - Groovy Script Prerequisite: Jenkins installed Up and running on your target master machine. Jenkins URL accessible   Step 1: Install Active Choice plugin On the Jenkins Dashboard, select the Manage Jenkins, Plugin- Manager, In the Available tab search for word 'Active', where you can see Active Choice plugin and choose installation option, and this will enables three different parameters in the "Add Paramters" list. They are : 1. Active choice parameter 2. Active Choice Reactive parameter 3. Active choice Reactive Reference parameter Here In my example I will use two of them, Firstly Active Choice Parameter for "environment". Create new item Name: active_project select a freestyle project click OK button. In the General tab, select the checkbo...
Read more