promql: faster range-query of label_replace and label_join by bboreham · Pull Request #13536 · prometheus/prometheus

bboreham · 2024-02-04T10:25:46Z

These functions act on the labels only, so don't need to go step by step over the samples in a range query.

I'm not entirely happy about the switch that calls the new version, but it is in line with some previous changes.
I think the whole of eval and evalRange should be refactored to make it easier to understand.

I left the previous entry-points in place with a panic, to make it obvious if someone is calling them in a way I didnt know about.

Benchmarks

goos: darwin
goarch: arm64
pkg: github.com/prometheus/prometheus/promql
                                                                               │  before.txt  │             after.txt              │
                                                                               │    sec/op    │   sec/op     vs base               │
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1-8           14.32µ ± 0%   13.43µ ± 1%   -6.28% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=100-8         35.54µ ± 0%   16.67µ ± 1%  -53.10% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1000-8       214.67µ ± 1%   39.39µ ± 1%  -81.65% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=1-8           35.11µ ± 1%   30.48µ ± 0%  -13.18% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=100-8        131.30µ ± 0%   61.87µ ± 1%  -52.88% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=1000-8        937.1µ ± 1%   285.1µ ± 1%  -69.57% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1-8       249.0µ ± 0%   206.7µ ± 2%  -16.98% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=100-8    1169.6µ ± 3%   513.7µ ± 0%  -56.07% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1000-8    8.847m ± 1%   2.715m ± 0%  -69.31% (p=0.002 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1-8                  12.21µ ± 1%   11.21µ ± 1%   -8.14% (p=0.002 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=100-8                38.33µ ± 1%   14.41µ ± 1%  -62.41% (p=0.002 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1000-8              263.34µ ± 1%   37.25µ ± 1%  -85.86% (p=0.002 n=6)
RangeQuery/expr=label_join(a_ten,_'l2',_'-',_'l',_'l'),steps=1-8                  31.94µ ± 1%   26.95µ ± 1%  -15.60% (p=0.002 n=6)
RangeQuery/expr=label_join(a_ten,_'l2',_'-',_'l',_'l'),steps=100-8               140.45µ ± 0%   58.11µ ± 2%  -58.63% (p=0.002 n=6)
RangeQuery/expr=label_join(a_ten,_'l2',_'-',_'l',_'l'),steps=1000-8              1060.0µ ± 1%   279.9µ ± 1%  -73.60% (p=0.002 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1-8              235.2µ ± 1%   188.4µ ± 1%  -19.89% (p=0.002 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=100-8           1236.9µ ± 1%   498.1µ ± 1%  -59.73% (p=0.002 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1000-8           9.784m ± 1%   2.696m ± 1%  -72.45% (p=0.002 n=6)
geomean                                                                           224.1µ        99.49µ       -55.60%

                                                                               │  before.txt   │              after.txt              │
                                                                               │     B/op      │     B/op      vs base               │
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1-8           13.41Ki ± 0%   12.11Ki ± 0%   -9.69% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=100-8         25.48Ki ± 0%   12.57Ki ± 0%  -50.67% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1000-8       135.62Ki ± 0%   17.16Ki ± 0%  -87.35% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=1-8           23.31Ki ± 0%   16.41Ki ± 0%  -29.59% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=100-8         38.26Ki ± 0%   19.75Ki ± 0%  -48.38% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=1000-8       173.86Ki ± 0%   49.82Ki ± 0%  -71.35% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1-8      142.74Ki ± 0%   61.16Ki ± 0%  -57.15% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=100-8    185.66Ki ± 0%   92.45Ki ± 0%  -50.21% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1000-8    569.3Ki ± 0%   370.4Ki ± 0%  -34.94% (p=0.002 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1-8                 10.602Ki ± 0%   9.240Ki ± 0%  -12.84% (p=0.002 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=100-8               28.860Ki ± 0%   9.695Ki ± 0%  -66.41% (p=0.002 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1000-8              195.28Ki ± 0%   14.29Ki ± 0%  -92.68% (p=0.002 n=6)
RangeQuery/expr=label_join(a_ten,_'l2',_'-',_'l',_'l'),steps=1-8                  20.25Ki ± 0%   13.13Ki ± 0%  -35.17% (p=0.002 n=6)
RangeQuery/expr=label_join(a_ten,_'l2',_'-',_'l',_'l'),steps=100-8                41.39Ki ± 0%   16.46Ki ± 0%  -60.23% (p=0.002 n=6)
RangeQuery/expr=label_join(a_ten,_'l2',_'-',_'l',_'l'),steps=1000-8              233.24Ki ± 0%   46.52Ki ± 0%  -80.06% (p=0.002 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1-8             137.22Ki ± 0%   52.47Ki ± 0%  -61.76% (p=0.002 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=100-8           186.31Ki ± 0%   83.82Ki ± 0%  -55.01% (p=0.002 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1000-8           626.1Ki ± 0%   361.7Ki ± 0%  -42.23% (p=0.002 n=6)
geomean                                                                           80.52Ki        32.68Ki       -59.42%

                                                                               │  before.txt  │             after.txt              │
                                                                               │  allocs/op   │  allocs/op   vs base               │
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1-8            281.0 ± 0%    260.0 ± 0%   -7.47% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=100-8          782.0 ± 0%    266.0 ± 0%  -65.98% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1000-8        5326.0 ± 0%    310.0 ± 0%  -94.18% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=1-8            387.0 ± 0%    347.0 ± 0%  -10.34% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=100-8          933.0 ± 0%    398.0 ± 0%  -57.34% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=1000-8        5846.0 ± 0%    811.0 ± 0%  -86.13% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1-8       1.478k ± 0%   1.251k ± 0%  -15.36% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=100-8     2.462k ± 0%   1.740k ± 0%  -29.33% (p=0.002 n=6)
RangeQuery/expr=label_replace(a_hundred,_'l2',_'$1',_'l',_'(.*)'),steps=1000-8   10.970k ± 0%   5.748k ± 0%  -47.60% (p=0.002 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1-8                   240.0 ± 0%    217.0 ± 0%   -9.58% (p=0.002 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=100-8                 939.0 ± 0%    223.0 ± 0%  -76.25% (p=0.002 n=6)
RangeQuery/expr=label_join(a_one,_'l2',_'-',_'l',_'l'),steps=1000-8               7283.0 ± 0%    267.0 ± 0%  -96.33% (p=0.002 n=6)
RangeQuery/expr=label_join(a_ten,_'l2',_'-',_'l',_'l'),steps=1-8                   336.0 ± 0%    293.0 ± 0%  -12.80% (p=0.002 n=6)
RangeQuery/expr=label_join(a_ten,_'l2',_'-',_'l',_'l'),steps=100-8                1080.0 ± 0%    344.0 ± 0%  -68.15% (p=0.002 n=6)
RangeQuery/expr=label_join(a_ten,_'l2',_'-',_'l',_'l'),steps=1000-8               7793.0 ± 0%    757.0 ± 0%  -90.29% (p=0.002 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1-8              1.247k ± 0%   1.016k ± 0%  -18.52% (p=0.002 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=100-8            2.429k ± 0%   1.505k ± 0%  -38.04% (p=0.002 n=6)
RangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=1000-8          12.736k ± 0%   5.512k ± 0%  -56.72% (p=0.002 n=6)
geomean                                                                           1.722k         630.7       -63.36%

These functions act on the labels only, so don't need to go step by step over the samples in a range query. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

beorn7

😎 Thank you very much.

juliusv · 2025-07-09T08:41:55Z

This seems to have broken the data + query case where series are not duplicate at individual steps of a range query, but become duplicates when combining the outputs of multiple independent range query steps, as reported in #15114

promql: faster range-query of label_replace and label_join

fdd5b85

These functions act on the labels only, so don't need to go step by step over the samples in a range query. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

bboreham requested a review from roidelapluie as a code owner February 4, 2024 10:25

promql: can now remove regex field from EvalNodeHelper

d3c1f0d

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

bboreham added the kind/optimization label Feb 20, 2024

beorn7 approved these changes Feb 29, 2024

View reviewed changes

beorn7 merged commit 9187bcb into prometheus:main Feb 29, 2024

alanprot mentioned this pull request Mar 4, 2024

Chore: Update Prometheus cortexproject/cortex#5795

Merged

3 tasks

bboreham mentioned this pull request Mar 23, 2024

[BUGFIX] PromQL: Validate label_join destination label #13803

Merged

bboreham mentioned this pull request Apr 2, 2024

Remove unused Dmn field on EvalNodeHelper #13877

Merged

juliusv mentioned this pull request Jul 9, 2025

label_replace does not merge series #15114

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

promql: faster range-query of label_replace and label_join#13536

promql: faster range-query of label_replace and label_join#13536
beorn7 merged 2 commits intoprometheus:mainfrom
bboreham:faster-label-replace

bboreham commented Feb 4, 2024 •

edited

Loading

Uh oh!

beorn7 left a comment

Uh oh!

juliusv commented Jul 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

bboreham commented Feb 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beorn7 left a comment

Choose a reason for hiding this comment

Uh oh!

juliusv commented Jul 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bboreham commented Feb 4, 2024 •

edited

Loading