Add query log by roidelapluie · Pull Request #6520 · prometheus/prometheus

roidelapluie · 2019-12-25T15:07:18Z

It is set in the configuration file to be changed online.

Fixes #1315

Signed-off-by: Julien Pivotto roidelapluie@inuits.eu

brian-brazil · 2019-12-25T16:44:59Z

Request logs still don't belong in application logs.

roidelapluie · 2019-12-26T22:17:57Z

As stated before in this project, we try not to reinvent the wheel.

Given that file rotation is usually a solved problem, I think we can:

Not do it ourselve
Reopen the log file upon every reload (that would align us with other softwares). That makes us compatible with e.g. logrotate.

brian-brazil · 2019-12-26T23:30:01Z

Reopen the log file upon every reload

That seems like a reasonable approach.

config/config.go

docs/configuration/configuration.md

promql/slow_query_logger.go

roidelapluie · 2019-12-27T21:56:57Z

Updated to remove the "slow" part ; now we will log all queries independently of their duration.
Added tests to check that we reopen the files and keep them open in case of reload failure
Issue a warning if we can't close previous file (that does not make reload fail)
Added comments in code

roidelapluie · 2019-12-27T21:58:05Z

I have also added the query log format to the list of things considered unstable so we have the ability to make the format evolve in the future.

roidelapluie · 2019-12-27T22:06:09Z

Further questions @brian-brazil:

Should we add a metric query_log_enabled 1 or 0 ? (could be done as a separated PR)

Should we log as INFO in the application log the slow query log activation / deactivation / filepath change ? (could be done as a separated PR)

brian-brazil · 2019-12-28T07:50:36Z

Should we log as INFO in the application log the slow query log activation / deactivation / filepath change ?

Sounds more debug to me offhand.

Should we add a metric query_log_enabled 1 or 0 ? (could be done as a separated PR)

Makes sense, this may have performance impact.

brian-brazil

Per the dev summit this should include IP addresses when it's coming from the query API.

docs/configuration/configuration.md

docs/stability.md

promql/engine.go

promql/query_logger.go

promql/engine.go

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

web/api/v1/api_test.go

promql/engine.go

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

roidelapluie · 2020-01-07T13:58:43Z

Exemple log line:
{"clientIP":"127.0.0.1","end":"2020-01-07T13:54:16.128Z","query":"vector(1)","start":"2020-01-07T13:54:16.128Z","stats":{"timings":{"evalTotalTime":0.000026106,"resultSortTime":0,"queryPreparationTime":0.000012256,"innerEvalTime":0.000007964,"execQueueTime":0.000002085,"execTotalTime":0.000041266}},"step":"0s","ts":"2020-01-07T13:54:16.160Z"}

With error:
{"clientIP":"127.0.0.1","end":"2020-01-07T13:56:19.132Z","error":"found duplicate series for the match group {} on the right hand-side of the operation: [{__name__=\"up\", instance=\"127.0.1.2:9090\", job=\"prometheus\"}, {__name__=\"up\", instance=\"127.0.1.1:9090\", job=\"prometheus\"}];many-to-many matching not allowed: matching labels must be unique on one side","query":"vector(9999) + on () up","start":"2020-01-07T13:56:19.132Z","stats":{"timings":{"evalTotalTime":0.000224988,"resultSortTime":0,"queryPreparationTime":0.000078192,"innerEvalTime":0,"execQueueTime":0.000003185,"execTotalTime":0.000239282}},"step":"0s","ts":"2020-01-07T13:56:19.169Z"}

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

roidelapluie · 2020-01-08T09:30:02Z

Should we use a dedupe logger for the errors? That would imply to add another logger to the engine, specifically for the query log errors.

brian-brazil · 2020-01-08T09:31:10Z

Dedupe logger loses us line numbers, and disk filling up is fatal to Prometheus unlike remote storage going away.

brian-brazil

Just some minor style stuff

promql/engine.go

promql/engine_test.go

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

brian-brazil · 2020-01-08T13:28:49Z

Thanks!

yeya24 · 2020-01-08T14:04:31Z

promql/engine.go

 }

+// SetQueryLogger sets the query logger.
+func (ng *Engine) SetQueryLogger(l QueryLogger) error {


Can we just drop this return value? Seems it only returns nil

docs/stability.md

brancz · 2020-01-09T07:54:39Z

docs/configuration/configuration.md


+  # File to which PromQL queries are logged.
+  # Reloading the configuration will reopen the file.
+  [ query_log_file: <string> ]


If I recall correctly we discussed at the devsummit that we should document how to allow people who want to to have this be part of the stdout log stream and have some docs around it. I assume that it's just a matter of setting this to /dev/stdout, I just wonder about stream syncronization.

Probably best for other docs. Does CON still exist on Windows?

Is it worth a complete section? under Configuration? under Querying?

Querying is for the PromQL language.

It wouldn't be part of the reference docs at all, this is generic Unix/Windows knowledge.

Should it be a small post on the blog, with example with logrotate etc ?

(blog post would be published after the release)

I think a blog with examples for logrotate and stdout sounds like a great idea, or a user guide.

roidelapluie · 2020-01-09T09:06:33Z

On 09 Jan 00:54, Frederic Branczyk wrote: I think a blog with examples for logrotate and stdout sounds like a great idea, or a user guide.

Oh yes a user guide sounds even better.

cstyan · 2020-02-07T00:02:02Z

/prombench v2.15.2

prombot · 2020-02-07T00:02:04Z

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-6520 and v2.15.2

After successful deployment, the benchmarking metrics can be viewed at:

Other Commands:
To stop benchmark: /prombench cancel
To restart benchmark: /prombench restart v2.12.0

cstyan · 2020-02-07T00:33:37Z

/prombench cancel

prombot · 2020-02-07T00:33:39Z

Benchmark cancel is in progress.

roidelapluie force-pushed the slowquery branch 4 times, most recently from 6d755d9 to 0983159 Compare December 25, 2019 16:37

roidelapluie force-pushed the slowquery branch from 0983159 to bfe8b66 Compare December 25, 2019 17:03

roidelapluie mentioned this pull request Dec 25, 2019

Add option to log slow queries and recording rules #1315

Closed

roidelapluie force-pushed the slowquery branch 5 times, most recently from 3f360f6 to c2aa86c Compare December 25, 2019 20:56

roidelapluie force-pushed the slowquery branch 3 times, most recently from 034b277 to 07e99d1 Compare December 26, 2019 22:52

brian-brazil reviewed Dec 27, 2019

View reviewed changes

config/config.go Outdated Show resolved Hide resolved

brian-brazil reviewed Dec 27, 2019

View reviewed changes

docs/configuration/configuration.md Outdated Show resolved Hide resolved

brian-brazil reviewed Dec 27, 2019

View reviewed changes

promql/slow_query_logger.go Outdated Show resolved Hide resolved

roidelapluie changed the title ~~Add option to log slow queries~~ Add query log Dec 27, 2019

roidelapluie force-pushed the slowquery branch from 2a70a3b to 0ba76ab Compare December 27, 2019 21:23

roidelapluie marked this pull request as ready for review December 27, 2019 21:26

roidelapluie force-pushed the slowquery branch 2 times, most recently from 1a15436 to 9ff502d Compare December 27, 2019 21:41

brian-brazil reviewed Dec 28, 2019

View reviewed changes

Add failures counter

93d7b2e

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

brian-brazil reviewed Jan 7, 2020

View reviewed changes

web/api/v1/api_test.go Show resolved Hide resolved

promql/engine.go Outdated Show resolved Hide resolved

promql/engine.go Outdated Show resolved Hide resolved

roidelapluie commented Jan 7, 2020

View reviewed changes

promql/engine.go Outdated Show resolved Hide resolved

Julien Pivotto added 4 commits January 7, 2020 14:17

Remove unneeded function

727a74e

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

Address err comments

cefb550

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

Remove test log and panic

0e9c513

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

Remove case

0d369b0

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

Use snake case to be consistent with timers

f2d82d9

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

roidelapluie requested review from brian-brazil and bwplotka January 8, 2020 09:41

brian-brazil reviewed Jan 8, 2020

View reviewed changes

promql/engine.go Outdated Show resolved Hide resolved

promql/engine_test.go Outdated Show resolved Hide resolved

Julien Pivotto added 2 commits January 8, 2020 14:14

Remove comment and re-scope queryIndex

bfc7edc

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

Add full stop on comments

2ff162e

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

brian-brazil merged commit 9d9bc52 into prometheus:master Jan 8, 2020

yeya24 reviewed Jan 8, 2020

View reviewed changes

brancz reviewed Jan 9, 2020

View reviewed changes

roidelapluie mentioned this pull request Jan 23, 2020

Proposal: Add query - client-IP pairs to prometheus logs #5846

Closed

prombot added the prombench label Feb 7, 2020

bchess mentioned this pull request Feb 15, 2020

Add support for query_log_file prometheus-operator/prometheus-operator#3025

Closed

This was referenced Feb 27, 2026

🌱 CNCF mission generation 2026-02-27 kubestellar/console-kb#6

Closed

🌱 CNCF mission generation 2026-02-27 kubestellar/console-kb#11

Merged

Conversation

roidelapluie commented Dec 25, 2019

Uh oh!

brian-brazil commented Dec 25, 2019

Uh oh!

roidelapluie commented Dec 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brian-brazil commented Dec 26, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roidelapluie commented Dec 27, 2019

Uh oh!

roidelapluie commented Dec 27, 2019

Uh oh!

roidelapluie commented Dec 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brian-brazil commented Dec 28, 2019

Uh oh!

brian-brazil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roidelapluie commented Jan 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roidelapluie commented Jan 8, 2020

Uh oh!

brian-brazil commented Jan 8, 2020

Uh oh!

brian-brazil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

brian-brazil commented Jan 8, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roidelapluie commented Jan 9, 2020 via email

Uh oh!

cstyan commented Feb 7, 2020

Uh oh!

prombot commented Feb 7, 2020

Uh oh!

cstyan commented Feb 7, 2020

Uh oh!

prombot commented Feb 7, 2020

Uh oh!

Reviewers

roidelapluie commented Dec 26, 2019 •

edited

Loading

roidelapluie commented Dec 27, 2019 •

edited

Loading

roidelapluie commented Jan 7, 2020 •

edited

Loading