add prometheus by marsishandsome · Pull Request #403 · pingcap/tiflash

marsishandsome · 2020-01-27T00:08:57Z

what this PR does

add prometheus-cpp as a third-party library
start a thread to send data from system.metrics, system.events and system.asynchronous_metrics tables to prometheus
support both pull mode and push mode

pull mode

add the following config to enable pull mode

[status]
# Prometheus metrics port, leaves it empty will disable prometheus pull.
metrics_port = 8234

push mode

add the following config to enable pull mode

[status]
# Prometheus pushgateway address, leaves it empty will disable prometheus push.
metrics_addr = "pushgateway:9091"

# Prometheus client push interval in second, min=5, max=120, default=15.
#metrics_interval = 15

how to add new metrics

#include <prometheus/gauge.h>
#include "MetricsPrometheus.h"

auto& gauge_family = prometheus::BuildGauge()
        .Name("time_running_seconds_total")
        .Help("How many seconds is this server running?")
        .Labels({{"label", "value"}})
        .Register(*DB::MetricsPrometheus::getRegistry());

auto& gauge_metric_1 = gauge_family.Add(
        {{"another_label", "value1"}, {"yet_another_label", "value1"}});

auto& gauge_metric_2 = gauge_family.Add(
        {{"another_label", "value2"}, {"yet_another_label", "value2"}});

while(true)
{
    std::this_thread::sleep_for(std::chrono::seconds(1));
    gauge_metric_1.Increment(1);
    gauge_metric_2.Increment(1);
}

how to get tiflash metrics (pull mode)

http://127.0.0.1:8234/metrics

# HELP tiflash_system_profile_events_Query Get from system.metrics, system.events and system.asynchronous_metrics tables
# TYPE tiflash_system_profile_events_Query gauge
tiflash_system_profile_events_Query 0.000000
# HELP tiflash_system_profile_events_SelectQuery Get from system.metrics, system.events and system.asynchronous_metrics tables
# TYPE tiflash_system_profile_events_SelectQuery gauge
tiflash_system_profile_events_SelectQuery 0.000000
# HELP tiflash_system_profile_events_InsertQuery Get from system.metrics, system.events and system.asynchronous_metrics tables
# TYPE tiflash_system_profile_events_InsertQuery gauge
tiflash_system_profile_events_InsertQuery 0.000000
# HELP tiflash_system_profile_events_DeleteQuery Get from system.metrics, system.events and system.asynchronous_metrics tables
# TYPE tiflash_system_profile_events_DeleteQuery gauge
tiflash_system_profile_events_DeleteQuery 0.000000
# HELP tiflash_system_profile_events_FileOpen Get from system.metrics, system.events and system.asynchronous_metrics tables
# TYPE tiflash_system_profile_events_FileOpen gauge
tiflash_system_profile_events_FileOpen 0.000000
# HELP tiflash_system_profile_events_FileOpenFailed Get from system.metrics, system.events and system.asynchronous_metrics tables
# TYPE tiflash_system_profile_events_FileOpenFailed gauge
tiflash_system_profile_events_FileOpenFailed 0.000000
# HELP tiflash_system_profile_events_Seek Get from system.metrics, system.events and system.asynchronous_metrics tables
# TYPE tiflash_system_profile_events_Seek gauge
tiflash_system_profile_events_Seek 0.000000

how to start prometheus

1. git clone git@github.com:marsishandsome/tiflash-docker-compose.git
2. cd tiflash-docker-compose
3. edit config/prometheus.yml, replace `marsishandsome.local` with your local hostname
4. docker-compose up -d

Grafana
http://127.0.0.1:3000/

Prometheus
http://127.0.0.1:9090/

Pushgateway
http://127.0.0.1:9091/

get tiflash metrics on Prometheus

get tiflash metrics on Grafana

marsishandsome · 2020-01-27T00:16:24Z

/build

marsishandsome · 2020-01-27T05:09:06Z

/run-integration-tests

marsishandsome · 2020-01-27T08:28:05Z

/run-integration-tests

ilovesoup · 2020-01-27T10:56:52Z

dbms/src/Server/MetricsPrometheus.cpp

+
+        auto pos = metricsAddr.find(':', 0);
+        auto host = metricsAddr.substr(0, pos);
+        auto port = metricsAddr.substr(pos + 1, metricsAddr.size());


Malformat on this crashes without useful information?

dbms/src/Server/MetricsPrometheus.cpp

ilovesoup · 2020-01-27T11:43:24Z

dbms/src/Server/MetricsPrometheus.cpp

+        if (metricsInterval > 0 && registry != nullptr)
+            break;
+
+        if (cond.wait_until(lock, get_next_time(5), [this] { return quit; }))


Not sure if a Poco::Timer suits you better here? At least it is simpler.
https://pocoproject.org/docs/Poco.Util.Timer.html

dbms/src/Server/MetricsPrometheus.cpp

ilovesoup · 2020-01-27T11:47:31Z

dbms/src/Server/MetricsPrometheus.cpp

+
+    if (gateway != nullptr)
+    {
+        auto returnCode = gateway->Push();


returnCode Var name style?

marsishandsome · 2020-01-28T03:36:26Z

/run-integration-tests

ilovesoup · 2020-01-28T08:11:16Z

dbms/src/Server/MetricsPrometheus.cpp

+    metrics_interval = conf.getInt(status_metrics_interval, 15);
+    if (metrics_interval < 5 || metrics_interval > 120)
+    {
+        metrics_interval = 15;


Usually capping should be to its closest number?

Also capping might should print a log.

ilovesoup

LGTM

marsishandsome · 2020-01-28T10:53:45Z

/build

marsishandsome · 2020-01-28T12:01:47Z

/build

marsishandsome · 2020-01-28T12:56:00Z

/run-integration-tests

marsishandsome · 2020-01-28T13:37:31Z

/run-integration-tests

marsishandsome · 2020-01-28T13:38:18Z

/run-integration-tests

marsishandsome added 2 commits January 26, 2020 20:47

add prometheus

48bad38

address code review

6ec8fab

marsishandsome force-pushed the send-metrics-to-Prometheus branch from 53c1a87 to 1666b2c Compare January 27, 2020 00:41

marsishandsome changed the title ~~Send metrics to prometheus~~ add prometheus Jan 27, 2020

marsishandsome force-pushed the send-metrics-to-Prometheus branch 2 times, most recently from ff74c62 to 3e34604 Compare January 27, 2020 07:01

add pull mode

ad0e8d3

marsishandsome force-pushed the send-metrics-to-Prometheus branch from 3e34604 to ad0e8d3 Compare January 27, 2020 07:04

ilovesoup requested review from flowbehappy, solotzg and zanmato1984 and removed request for zanmato1984 January 27, 2020 08:45

ilovesoup suggested changes Jan 27, 2020

View reviewed changes

marsishandsome force-pushed the send-metrics-to-Prometheus branch from 8b645c2 to be03297 Compare January 28, 2020 01:04

address code review

68ef837

use timer

a164c00

marsishandsome force-pushed the send-metrics-to-Prometheus branch from be03297 to a164c00 Compare January 28, 2020 08:01

ilovesoup reviewed Jan 28, 2020

View reviewed changes

ilovesoup approved these changes Jan 28, 2020

View reviewed changes

address code review

b2f3860

marsishandsome force-pushed the send-metrics-to-Prometheus branch from 60e5adb to b2f3860 Compare January 28, 2020 08:34

marsishandsome merged commit d01d69a into master Jan 28, 2020

solotzg deleted the send-metrics-to-Prometheus branch February 7, 2020 03:54

CalvinNeo mentioned this pull request Dec 16, 2024

update raftstore-proxy to eliminate the forced wait 10s when profiling #9730

Merged

12 tasks

CalvinNeo mentioned this pull request Jan 9, 2025

Set proxy's memory limit by TiFlash (#9753) and update tiflash proxy #9776

Merged

12 tasks

Conversation

marsishandsome commented Jan 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

what this PR does

pull mode

push mode

how to add new metrics

how to get tiflash metrics (pull mode)

how to start prometheus

get tiflash metrics on Prometheus

get tiflash metrics on Grafana

Uh oh!

marsishandsome commented Jan 27, 2020

Uh oh!

marsishandsome commented Jan 27, 2020

Uh oh!

marsishandsome commented Jan 27, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marsishandsome commented Jan 28, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilovesoup left a comment

Choose a reason for hiding this comment

Uh oh!

marsishandsome commented Jan 28, 2020

Uh oh!

marsishandsome commented Jan 28, 2020

Uh oh!

marsishandsome commented Jan 28, 2020

Uh oh!

marsishandsome commented Jan 28, 2020

Uh oh!

marsishandsome commented Jan 28, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

marsishandsome commented Jan 27, 2020 •

edited

Loading