Skip to content

Commit fd1f3e8

Browse files
jaydelucazeitlinger
jaydeluca
and
zeitlinger
authored
feat: Add custom labels to exemplars (#2191)
Resolves #1994 --------- Signed-off-by: Jay DeLuca <jaydeluca4@gmail.com> Co-authored-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
1 parent 9e1f2f5 commit fd1f3e8

10 files changed

Lines changed: 460 additions & 7 deletions

File tree

‎docs/apidiffs/current_vs_latest/prometheus-metrics-core.txt‎

Lines changed: 18 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/content/otel/tracing.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,3 +75,43 @@ The [examples/example-exemplar-tail-sampling/](https://github.com/prometheus/cli
7575
directory has a complete end-to-end example, with a distributed Java application with two services,
7676
an OpenTelemetry collector, Prometheus, Tempo as a trace database, and Grafana dashboards. Use
7777
docker-compose as described in the example's readme to run the example and explore the results.
78+
79+
## Adding custom labels to exemplars
80+
81+
Automatically-sampled exemplars carry the `trace_id` and `span_id` labels. You can attach
82+
additional, custom labels (for example an internal identifier) to every automatically-sampled
83+
exemplar. There are two options.
84+
85+
### Global (all metrics)
86+
87+
Register a global supplier to add custom labels to the exemplars of _all_ metrics, including
88+
metrics registered by third-party libraries that you do not control. This is the right option when
89+
you cannot modify the code that creates the metric:
90+
91+
```java
92+
ExemplarLabelsSupplier.setExemplarLabelsSupplier(
93+
() -> Labels.of("management_id", currentManagementId()));
94+
```
95+
96+
### Per metric
97+
98+
If you only want the extra labels on a specific metric you define yourself, use the builder:
99+
100+
```java
101+
Counter counter =
102+
Counter.builder()
103+
.name("requests_total")
104+
.exemplarLabelsSupplier(() -> Labels.of("management_id", currentManagementId()))
105+
.build();
106+
```
107+
108+
### Notes
109+
110+
- The supplier is invoked on the (rate-limited) hot path each time an exemplar is sampled, so it
111+
should be cheap. It may return dynamic, request-scoped values (e.g. read from a thread-local).
112+
- Custom labels are only added when a valid, sampled span context is present; the supplier never
113+
causes an exemplar to be created on its own.
114+
- Precedence on a label-name collision: the reserved `trace_id`/`span_id` labels always win, then
115+
the per-metric supplier, then the global supplier. Colliding labels are silently dropped.
116+
- If the supplier throws, the exception is swallowed and the exemplar is created without the
117+
additional labels, so a misbehaving supplier never breaks metric collection.

‎prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/exemplars/ExemplarLabelsSupplier.java‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
package io.prometheus.metrics.core.exemplars;
2+
3+
import io.prometheus.metrics.annotations.StableApi;
4+
import io.prometheus.metrics.model.snapshots.Labels;
5+
import java.util.concurrent.atomic.AtomicReference;
6+
import java.util.function.Supplier;
7+
import javax.annotation.Nullable;
8+
9+
/**
10+
* Global holder for a {@link Supplier} of additional {@link Labels} that are merged into every
11+
* automatically-sampled Exemplar across the entire application.
12+
*
13+
* <p>This is the global counterpart to the per-metric {@code exemplarLabelsSupplier(...)} builder
14+
* method. Registering a supplier here affects <em>all</em> metrics, including metrics registered by
15+
* third-party libraries that the application does not control. This makes it the right tool when
16+
* you cannot modify the code that creates the metrics.
17+
*
18+
* <p>The supplier is invoked on the metric hot path (rate-limited by the exemplar sampler), each
19+
* time an Exemplar is sampled from a valid, sampled span context. It should therefore be cheap and
20+
* non-blocking. It may return dynamic, request-scoped values, for example an identifier read from a
21+
* thread-local:
22+
*
23+
* <pre>{@code
24+
* ExemplarLabelsSupplier.setExemplarLabelsSupplier(
25+
* () -> Labels.of("management_id", currentManagementId()));
26+
* }</pre>
27+
*
28+
* <p>Labels returned by the supplier that collide with {@code trace_id}/{@code span_id} (or, when a
29+
* per-metric supplier is also configured, with that supplier's labels) are silently dropped rather
30+
* than causing an error: the per-metric supplier takes precedence over the global one, and the
31+
* reserved {@code trace_id}/{@code span_id} labels always win. If the supplier throws, the
32+
* exception is swallowed and the Exemplar is created without the additional labels, so a
33+
* misbehaving supplier never breaks metric collection.
34+
*/
35+
@StableApi
36+
public class ExemplarLabelsSupplier {
37+
38+
private static final AtomicReference<Supplier<Labels>> supplierRef = new AtomicReference<>();
39+
40+
private ExemplarLabelsSupplier() {}
41+
42+
/**
43+
* Register a global supplier of additional exemplar labels. Pass {@code null} to remove a
44+
* previously registered supplier. The most recently registered supplier wins.
45+
*/
46+
public static void setExemplarLabelsSupplier(@Nullable Supplier<Labels> supplier) {
47+
supplierRef.set(supplier);
48+
}
49+
50+
/** Returns the registered global supplier, or {@code null} if none has been set. */
51+
@Nullable
52+
public static Supplier<Labels> getExemplarLabelsSupplier() {
53+
return supplierRef.get();
54+
}
55+
}

‎prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/exemplars/ExemplarSampler.java‎

Lines changed: 96 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
import java.util.concurrent.TimeUnit;
1414
import java.util.concurrent.atomic.AtomicBoolean;
1515
import java.util.function.LongSupplier;
16+
import java.util.function.Supplier;
1617
import javax.annotation.Nullable;
1718

1819
/**
@@ -47,8 +48,10 @@ public class ExemplarSampler {
4748
private final SpanContext
4849
spanContext; // may be null, in that case SpanContextSupplier.getSpanContext() is used.
4950

51+
@Nullable private final Supplier<Labels> additionalLabelsSupplier;
52+
5053
public ExemplarSampler(ExemplarSamplerConfig config) {
51-
this(config, null);
54+
this(config, null, null);
5255
}
5356

5457
/**
@@ -60,10 +63,24 @@ public ExemplarSampler(ExemplarSamplerConfig config) {
6063
* SpanContextSupplier.getSpanContext()} is called to find a span context.
6164
*/
6265
public ExemplarSampler(ExemplarSamplerConfig config, @Nullable SpanContext spanContext) {
66+
this(config, spanContext, null);
67+
}
68+
69+
/**
70+
* Constructor that additionally accepts a supplier of labels to be merged into every
71+
* automatically-sampled exemplar. The supplier is called each time an exemplar is sampled from a
72+
* span context, so it can return dynamic values (e.g. a request-scoped identifier). The supplier
73+
* is only called when a valid, sampled span context is present.
74+
*/
75+
public ExemplarSampler(
76+
ExemplarSamplerConfig config,
77+
@Nullable SpanContext spanContext,
78+
@Nullable Supplier<Labels> additionalLabelsSupplier) {
6379
this.config = config;
6480
this.exemplars = new Exemplar[config.getNumberOfExemplars()];
6581
this.customExemplars = new Exemplar[exemplars.length];
6682
this.spanContext = spanContext;
83+
this.additionalLabelsSupplier = additionalLabelsSupplier;
6784
}
6885

6986
public Exemplars collect() {
@@ -322,7 +339,7 @@ private long durationUntilNextExemplarExpires(long now) {
322339

323340
private long updateCustomExemplar(int index, double value, Labels labels, long now) {
324341
if (!labels.contains(Exemplar.TRACE_ID) && !labels.contains(Exemplar.SPAN_ID)) {
325-
labels = labels.merge(doSampleExemplar());
342+
labels = mergeLabels(labels, sampleTraceContextLabels());
326343
}
327344
customExemplars[index] =
328345
Exemplar.builder().value(value).labels(labels).timestampMillis(now).build();
@@ -341,6 +358,19 @@ private long updateExemplar(int index, double value, long now) {
341358
}
342359

343360
private Labels doSampleExemplar() {
361+
Labels labels = sampleTraceContextLabels();
362+
if (labels.isEmpty()) {
363+
return labels;
364+
}
365+
// Per-metric supplier first (more specific), then the global supplier. On a name
366+
// collision the earlier (more specific) value is kept; the reserved trace_id/span_id
367+
// labels always win over both.
368+
labels = mergeAdditionalLabels(labels, additionalLabelsSupplier);
369+
labels = mergeAdditionalLabels(labels, ExemplarLabelsSupplier.getExemplarLabelsSupplier());
370+
return labels;
371+
}
372+
373+
private Labels sampleTraceContextLabels() {
344374
// Using the qualified name so that Micrometer can exclude the dependency on
345375
// prometheus-metrics-tracer-initializer
346376
// as they provide their own implementation of SpanContextSupplier.
@@ -366,4 +396,68 @@ private Labels doSampleExemplar() {
366396
}
367397
return Labels.EMPTY;
368398
}
399+
400+
/**
401+
* Merge labels from {@code supplier} into {@code base}, dropping any label whose name already
402+
* exists in {@code base}. Never throws: a {@code null} supplier, a {@code null}/empty result, a
403+
* colliding label name, or an exception thrown by the supplier all result in {@code base} being
404+
* returned unchanged (minus the offending labels). A misbehaving supplier must never break metric
405+
* collection.
406+
*/
407+
private static Labels mergeAdditionalLabels(Labels base, @Nullable Supplier<Labels> supplier) {
408+
if (supplier == null) {
409+
return base;
410+
}
411+
Labels extra;
412+
try {
413+
extra = supplier.get();
414+
} catch (Throwable ignored) {
415+
// A misbehaving supplier (any RuntimeException or Error) must never break metric collection.
416+
return base;
417+
}
418+
if (extra == null || extra.isEmpty()) {
419+
return base;
420+
}
421+
return mergeLabels(base, extra);
422+
}
423+
424+
/**
425+
* Merge {@code extra} into {@code base}, dropping any label whose name already exists in {@code
426+
* base}.
427+
*/
428+
private static Labels mergeLabels(Labels base, Labels extra) {
429+
if (extra.isEmpty()) {
430+
return base;
431+
}
432+
// Count name collisions with base in a single pass so we can merge exactly once below: base
433+
// (trace_id/span_id and any more-specific supplier) always wins, so colliding labels are
434+
// dropped. extra is itself a valid Labels (no internal duplicates), so the surviving labels
435+
// never collide with each other and merge() cannot throw on a duplicate name.
436+
int size = extra.size();
437+
int collisions = 0;
438+
for (int i = 0; i < size; i++) {
439+
if (base.contains(extra.getName(i))) {
440+
collisions++;
441+
}
442+
}
443+
if (collisions == 0) {
444+
return base.merge(extra);
445+
}
446+
if (collisions == size) {
447+
return base;
448+
}
449+
int kept = size - collisions;
450+
String[] names = new String[kept];
451+
String[] values = new String[kept];
452+
int j = 0;
453+
for (int i = 0; i < size; i++) {
454+
String name = extra.getName(i);
455+
if (!base.contains(name)) {
456+
names[j] = name;
457+
values[j] = extra.getValue(i);
458+
j++;
459+
}
460+
}
461+
return base.merge(names, values);
462+
}
369463
}

‎prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Counter.java‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
import java.util.List;
1616
import java.util.concurrent.atomic.DoubleAdder;
1717
import java.util.concurrent.atomic.LongAdder;
18+
import java.util.function.Supplier;
1819
import javax.annotation.Nullable;
1920

2021
/**
@@ -37,6 +38,7 @@ public class Counter extends StatefulMetric<CounterDataPoint, Counter.DataPoint>
3738
implements CounterDataPoint {
3839

3940
@Nullable private final ExemplarSamplerConfig exemplarSamplerConfig;
41+
@Nullable private final Supplier<Labels> exemplarLabelsSupplier;
4042

4143
private Counter(Builder builder, PrometheusProperties prometheusProperties) {
4244
super(builder);
@@ -49,6 +51,7 @@ private Counter(Builder builder, PrometheusProperties prometheusProperties) {
4951
} else {
5052
exemplarSamplerConfig = null;
5153
}
54+
exemplarLabelsSupplier = builder.exemplarLabelsSupplier;
5255
}
5356

5457
@Override
@@ -108,7 +111,8 @@ public MetricType getMetricType() {
108111
@Override
109112
protected DataPoint newDataPoint() {
110113
if (exemplarSamplerConfig != null) {
111-
return new DataPoint(new ExemplarSampler(exemplarSamplerConfig));
114+
return new DataPoint(
115+
new ExemplarSampler(exemplarSamplerConfig, null, exemplarLabelsSupplier));
112116
} else {
113117
return new DataPoint(null);
114118
}

‎prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Gauge.java‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
import java.util.Collections;
1515
import java.util.List;
1616
import java.util.concurrent.atomic.AtomicLong;
17+
import java.util.function.Supplier;
1718
import javax.annotation.Nullable;
1819

1920
/**
@@ -44,6 +45,7 @@ public class Gauge extends StatefulMetric<GaugeDataPoint, Gauge.DataPoint>
4445
implements GaugeDataPoint {
4546

4647
@Nullable private final ExemplarSamplerConfig exemplarSamplerConfig;
48+
@Nullable private final Supplier<Labels> exemplarLabelsSupplier;
4749

4850
private Gauge(Builder builder, PrometheusProperties prometheusProperties) {
4951
super(builder);
@@ -56,6 +58,7 @@ private Gauge(Builder builder, PrometheusProperties prometheusProperties) {
5658
} else {
5759
exemplarSamplerConfig = null;
5860
}
61+
exemplarLabelsSupplier = builder.exemplarLabelsSupplier;
5962
}
6063

6164
@Override
@@ -110,7 +113,8 @@ public MetricType getMetricType() {
110113
@Override
111114
protected DataPoint newDataPoint() {
112115
if (exemplarSamplerConfig != null) {
113-
return new DataPoint(new ExemplarSampler(exemplarSamplerConfig));
116+
return new DataPoint(
117+
new ExemplarSampler(exemplarSamplerConfig, null, exemplarLabelsSupplier));
114118
} else {
115119
return new DataPoint(null);
116120
}

‎prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Histogram.java‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
import java.util.concurrent.atomic.AtomicBoolean;
2727
import java.util.concurrent.atomic.DoubleAdder;
2828
import java.util.concurrent.atomic.LongAdder;
29+
import java.util.function.Supplier;
2930
import javax.annotation.Nullable;
3031

3132
/**
@@ -73,6 +74,7 @@ public class Histogram extends StatefulMetric<DistributionDataPoint, Histogram.D
7374
private static final double[][] NATIVE_BOUNDS;
7475

7576
@Nullable private final ExemplarSamplerConfig exemplarSamplerConfig;
77+
@Nullable private final Supplier<Labels> exemplarLabelsSupplier;
7678

7779
// Upper bounds for the classic histogram buckets. Contains at least +Inf.
7880
// An empty array indicates that this is a native histogram only.
@@ -171,6 +173,7 @@ private Histogram(Histogram.Builder builder, PrometheusProperties prometheusProp
171173
} else {
172174
exemplarSamplerConfig = null;
173175
}
176+
exemplarLabelsSupplier = builder.exemplarLabelsSupplier;
174177
}
175178

176179
@Override
@@ -212,7 +215,7 @@ public class DataPoint implements DistributionDataPoint {
212215

213216
private DataPoint() {
214217
if (exemplarSamplerConfig != null) {
215-
exemplarSampler = new ExemplarSampler(exemplarSamplerConfig);
218+
exemplarSampler = new ExemplarSampler(exemplarSamplerConfig, null, exemplarLabelsSupplier);
216219
} else {
217220
exemplarSampler = null;
218221
}

0 commit comments

Comments
 (0)