Table of Contents
Why this post?
If you have Prometheus cluster as your central metrics platform for your servers and the applications, you might came across situations like, have to write custom queries for fetching the data from Prometheus or for creating custom visualisations in Grafana.
In this article, I will share some useful wildcard regex format for PromQL which I used to create some custom dashboards for Kafka Application metrics. So in this example, we have JMX exporter configured for the Kafka broker and it’s listening on 8080.
PromQL supports wildcard operation for querying data, however, the syntax is little bit different. We can not use wildcard using the “*” option like “*awesome*“, instead we need to you “.+” for wildcard operations.
Use of Tilde
For matching REGEX we need to use “~” and the string and wildcard regex inside double quote.
It is also possible to negatively match a label value, or to match label values against regular expressions. The following label matching operators exist:
=: Select labels that are exactly equal to the provided string.!=: Select labels that are not equal to the provided string.=~: Select labels that regex-match the provided string.!~: Select labels that do not regex-match the provided string.
Some sample labels
Consider the example below (a part of data in Prometheus):
{cluster="kafka",env="production",job="kafka",topic="_confluent-controlcenter-5-3-2-1-TriggerEventsStore-changelog"} 0
{cluster="kafka",env="production",job="kafka",topic="_confluent-controlcenter-5-3-2-1-expected-group-consumption-rekey"} 0
{cluster="kafka",env="production",job="kafka",topic="_confluent-ksql-default__command_topic"} 0
{cluster="kafka",env="production",job="kafka",topic="obkafkadb.public.product"} 0
{cluster="kafka",env="production",job="kafka",topic="lebron-KSTREAM-MAP-0000000009-repartition"} 0
{cluster="kafka",env="production",job="kafka",topic="_confluent-controlcenter-5-3-2-1-MonitoringStream-THREE_HOURS-changelog"} 0
These are few entries that fetched using the following query:
sum without(instance)(rate(kafka_server_brokertopicmetrics_messagesin_total{job="kafka",topic!=""}[5m]))
This actually fetch the “Messages In” on all the topics. Here we’re using the matching operator “!=” and followed by no specific topic name. So it will do an inverse match and list out all topics. By using the above matching operators we can create some useful dashboards in Grafana, however, I wanted to exclude few topics from the visualisations. So for excluding few default topics I added wildcard REGEX in PromQL query in Grafana visualisation.
For example, if we want to exclude all topics which starting with “_confluent” we can use the following wildcard REGEX in our query:
topic!~".+confluent.+"
The complete query looks like:
sum without(instance)(rate(kafka_server_brokertopicmetrics_messagesin_total{job="kafka",topic!~".+confluent.+"}[5m]))
If you want to use multiple wildcards, you can add those as comma separated.
sum without(instance)(rate(kafka_server_brokertopicmetrics_messagesin_total{job="kafka",topic!~".+confluent.+",topic!~".+consumer.+",topic!~"connect.+",topic!~".+schemas"}[5m]))
You can try different combinations as per your actual requirement.
Let me know if you have any questions. Will try to help you!!
