Use a DropReservoir when an exemplar.AlwaysOffFilter is provided by dashpole · Pull Request #8211 · open-telemetry/opentelemetry-go

dashpole · 2026-04-15T15:47:10Z

The spec for always off suggests this should "disable" the Exemplar feature:

An ExemplarFilter which makes no measurements eligible for being an Exemplar.
Using this ExemplarFilter is as good as disabling the Exemplar feature.

There were a few reports of much higher memory usage when exemplars were added. I looked into the reports, and it looks like exemplar reservoirs are expensive in terms of memory compared to the aggregation itself. Ideally, we shouldn't construct the reservoir when an AlwaysOff exemplar filter is used.

I couldn't find a super clean way to do this... exemplar.AlwaysOffFilter is a function, so I couldn't easily compare it without comparing the pointers. I'm open to other suggestions. The internet says that compiler optimizations could cause this logic to fail, but worst-case you get the existing behavior.

Benchmarks

I needed a benchmark that creates a new attribute set for each call to be able to see how much memory each attribute set uses, so I added a new one, and ran it against main.

Overall, it reduces the amount of memory used per-attribute-set by 70-88%, which is pretty substantial.

Overall,

                                                  │   old.txt   │               new.txt               │
                                                  │   sec/op    │    sec/op     vs base               │
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24            3.523µ ± 2%   3.745µ ± 29%   +6.32% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24          3.714µ ± 6%   3.879µ ±  5%        ~ (p=0.132 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24      3.811µ ± 8%   3.941µ ±  5%        ~ (p=0.065 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24    3.714µ ± 5%   3.822µ ±  1%        ~ (p=0.132 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24          3.126µ ± 4%   3.333µ ±  4%   +6.61% (p=0.004 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24        3.099µ ± 4%   3.204µ ±  8%        ~ (p=0.093 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24              3.745µ ± 7%   3.902µ ±  4%        ~ (p=0.240 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24            3.746µ ± 7%   3.862µ ±  2%        ~ (p=0.102 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24           3.621µ ± 3%   1.665µ ±  1%  -54.00% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24         3.639µ ± 2%   1.686µ ±  4%  -53.68% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24     3.563µ ± 4%   1.700µ ±  4%  -52.29% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24   3.634µ ± 1%   1.690µ ±  4%  -53.49% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24         2.892µ ± 2%   2.005µ ±  2%  -30.66% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24       2.962µ ± 5%   2.057µ ±  6%  -30.55% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24             3.692µ ± 5%   1.599µ ±  2%  -56.68% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24           3.651µ ± 3%   1.612µ ±  2%  -55.86% (p=0.002 n=6)
geomean                                             3.495µ        2.541µ        -27.30%

                                                  │   old.txt    │               new.txt               │
                                                  │     B/op     │     B/op      vs base               │
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24            3.107Ki ± 0%   3.107Ki ± 0%        ~ (p=0.121 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24          3.108Ki ± 0%   3.107Ki ± 0%        ~ (p=0.177 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24      3.108Ki ± 0%   3.108Ki ± 0%        ~ (p=0.675 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24    3.107Ki ± 0%   3.107Ki ± 0%        ~ (p=1.000 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24          2.622Ki ± 0%   2.622Ki ± 0%        ~ (p=0.394 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24        2.622Ki ± 0%   2.622Ki ± 0%        ~ (p=0.636 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24              3.108Ki ± 0%   3.107Ki ± 0%        ~ (p=0.167 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24            3.107Ki ± 0%   3.108Ki ± 0%        ~ (p=0.182 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24            3182.0 ± 0%     361.0 ± 0%  -88.65% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24          3182.5 ± 0%     360.5 ± 0%  -88.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24      3182.0 ± 0%     359.5 ± 1%  -88.70% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24    3182.0 ± 0%     361.0 ± 0%  -88.65% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24          2684.0 ± 0%     773.0 ± 0%  -71.20% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24        2684.0 ± 0%     773.0 ± 0%  -71.20% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24              3182.0 ± 0%     361.0 ± 1%  -88.65% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24            3182.0 ± 0%     361.0 ± 0%  -88.65% (p=0.002 n=6)
geomean                                             2.978Ki        1.127Ki       -62.17%

                                                  │   old.txt   │               new.txt               │
                                                  │  allocs/op  │ allocs/op   vs base                 │
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24             12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24           12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24       12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24     12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24           13.00 ± 0%   13.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24         13.00 ± 0%   13.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24               12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24             12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24           12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24         12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24     12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24   12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24         13.000 ± 0%   9.000 ± 0%  -30.77% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24       13.000 ± 0%   9.000 ± 0%  -30.77% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24             12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24           12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
geomean                                              12.24        9.553       -21.97%
¹ all samples are equal

Gemini helped me write this.

codecov · 2026-04-15T15:49:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.7%. Comparing base (8365fb9) to head (dda4983).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #8211   +/-   ##
=====================================
  Coverage   83.7%   83.7%           
=====================================
  Files        314     314           
  Lines      29727   29730    +3     
=====================================
+ Hits       24905   24910    +5     
+ Misses      4448    4447    -1     
+ Partials     374     373    -1

Files with missing lines	Coverage Δ
sdk/metric/exemplar.go	`100.0% <100.0%> (ø)`
sdk/metric/internal/aggregate/aggregate.go	`100.0% <100.0%> (ø)`
sdk/metric/internal/aggregate/drop.go	`100.0% <100.0%> (ø)`

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR reduces metric SDK memory usage when exemplars are effectively disabled by configuring exemplar.AlwaysOffFilter, by avoiding construction of exemplar reservoirs and using a drop reservoir instead. This aligns behavior more closely with the spec guidance that AlwaysOff is “as good as disabling” exemplars.

Changes:

Export and use an internal DropReservoir implementation as the default exemplar reservoir factory.
Update reservoirFunc to return DropReservoir when exemplar.AlwaysOffFilter is provided (via function pointer comparison).
Add a targeted test and a new benchmark to quantify allocations for unique attribute sets with exemplars on/off.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
sdk/metric/internal/aggregate/drop.go	Export `DropReservoir` factory for a reservoir that drops all offers.
sdk/metric/internal/aggregate/drop_test.go	Update test to call `DropReservoir`.
sdk/metric/internal/aggregate/aggregate.go	Use `DropReservoir` as the default reservoir factory when none is provided.
sdk/metric/internal/aggregate/aggregate_test.go	Update test helper to call `DropReservoir`.
sdk/metric/exemplar.go	Return `aggregate.DropReservoir` when configured filter is `exemplar.AlwaysOffFilter`.
sdk/metric/exemplar_test.go	Add test validating the AlwaysOff optimization path.
sdk/metric/benchmark_test.go	Add benchmark that creates a new attribute set per call for multiple instrument types and filters.
CHANGELOG.md	Add changelog entry for the optimization.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Found during copilot code reviews on other PRs. #8211 (comment)

Found during copilot code reviews on other PRs. open-telemetry#8211 (comment)

…n-telemetry#8211) Fixes open-telemetry#6260, open-telemetry#6333 The [spec for always off](https://github.com/open-telemetry/opentelemetry-specification/blob/d500678e4612b56ff2cd5f03e67cd845977d1746/specification/metrics/sdk.md#alwaysoff) suggests this should "disable" the Exemplar feature: > An ExemplarFilter which makes no measurements eligible for being an Exemplar. Using this ExemplarFilter is as good as disabling the Exemplar feature. There were a few reports of much higher memory usage when exemplars were added. I looked into the reports, and it looks like exemplar reservoirs are expensive in terms of memory compared to the aggregation itself. Ideally, we shouldn't construct the reservoir when an AlwaysOff exemplar filter is used. I couldn't find a super clean way to do this... `exemplar.AlwaysOffFilter` is a function, so I couldn't easily compare it without comparing the pointers. I'm open to other suggestions. The internet says that compiler optimizations could cause this logic to fail, but worst-case you get the existing behavior. ### Benchmarks I needed a benchmark that creates a new attribute set for each call to be able to see how much memory each attribute set uses, so I added a new one, and ran it against main. Overall, it reduces the amount of memory used per-attribute-set by 70-88%, which is pretty substantial. Overall, ``` │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24 3.523µ ± 2% 3.745µ ± 29% +6.32% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24 3.714µ ± 6% 3.879µ ± 5% ~ (p=0.132 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24 3.811µ ± 8% 3.941µ ± 5% ~ (p=0.065 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24 3.714µ ± 5% 3.822µ ± 1% ~ (p=0.132 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24 3.126µ ± 4% 3.333µ ± 4% +6.61% (p=0.004 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24 3.099µ ± 4% 3.204µ ± 8% ~ (p=0.093 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24 3.745µ ± 7% 3.902µ ± 4% ~ (p=0.240 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24 3.746µ ± 7% 3.862µ ± 2% ~ (p=0.102 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24 3.621µ ± 3% 1.665µ ± 1% -54.00% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24 3.639µ ± 2% 1.686µ ± 4% -53.68% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24 3.563µ ± 4% 1.700µ ± 4% -52.29% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24 3.634µ ± 1% 1.690µ ± 4% -53.49% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24 2.892µ ± 2% 2.005µ ± 2% -30.66% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24 2.962µ ± 5% 2.057µ ± 6% -30.55% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24 3.692µ ± 5% 1.599µ ± 2% -56.68% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24 3.651µ ± 3% 1.612µ ± 2% -55.86% (p=0.002 n=6) geomean 3.495µ 2.541µ -27.30% │ old.txt │ new.txt │ │ B/op │ B/op vs base │ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24 3.107Ki ± 0% 3.107Ki ± 0% ~ (p=0.121 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24 3.108Ki ± 0% 3.107Ki ± 0% ~ (p=0.177 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24 3.108Ki ± 0% 3.108Ki ± 0% ~ (p=0.675 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24 3.107Ki ± 0% 3.107Ki ± 0% ~ (p=1.000 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24 2.622Ki ± 0% 2.622Ki ± 0% ~ (p=0.394 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24 2.622Ki ± 0% 2.622Ki ± 0% ~ (p=0.636 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24 3.108Ki ± 0% 3.107Ki ± 0% ~ (p=0.167 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24 3.107Ki ± 0% 3.108Ki ± 0% ~ (p=0.182 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24 3182.0 ± 0% 361.0 ± 0% -88.65% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24 3182.5 ± 0% 360.5 ± 0% -88.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24 3182.0 ± 0% 359.5 ± 1% -88.70% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24 3182.0 ± 0% 361.0 ± 0% -88.65% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24 2684.0 ± 0% 773.0 ± 0% -71.20% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24 2684.0 ± 0% 773.0 ± 0% -71.20% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24 3182.0 ± 0% 361.0 ± 1% -88.65% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24 3182.0 ± 0% 361.0 ± 0% -88.65% (p=0.002 n=6) geomean 2.978Ki 1.127Ki -62.17% │ old.txt │ new.txt │ │ allocs/op │ allocs/op vs base │ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24 13.00 ± 0% 13.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24 13.00 ± 0% 13.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24 13.000 ± 0% 9.000 ± 0% -30.77% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24 13.000 ± 0% 9.000 ± 0% -30.77% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) geomean 12.24 9.553 -21.97% ¹ all samples are equal ``` Gemini helped me write this. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Follow-up to #8211. Because the exemplar reservoir is an interface, it can't be in-lined even when it is a no-op, so it has significant overhead. Skip it when a drop reservoir is used. This seems to have an overhead of something like ~10ns, which shows up mostly on the precomputed benchmarks. Benchmarks ``` goarch: amd64 pkg: go.opentelemetry.io/otel/sdk/metric cpu: Intel(R) Xeon(R) CPU @ 2.20GHz │ benchmark_results_before.txt │ benchmark_results_after.txt │ │ sec/op │ sec/op vs base │ EndToEndCounterAdd/NoFilter/Attributes/1/Precomputed/WithAttributeSet-24 62.65n ± 15% 54.59n ± 12% -12.88% (p=0.026 n=6) EndToEndCounterAdd/NoFilter/Attributes/1/Precomputed/WithAttributes-24 63.67n ± 10% 52.21n ± 6% -17.99% (p=0.002 n=6) EndToEndCounterAdd/NoFilter/Attributes/1/Dynamic/WithAttributeSet-24 238.8n ± 14% 240.0n ± 10% ~ (p=1.000 n=6) EndToEndCounterAdd/NoFilter/Attributes/1/Dynamic/WithAttributes-24 220.7n ± 14% 245.6n ± 13% ~ (p=0.180 n=6) EndToEndCounterAdd/NoFilter/Attributes/1/Naive/WithAttributes-24 373.4n ± 26% 352.9n ± 11% ~ (p=0.240 n=6) EndToEndCounterAdd/NoFilter/Attributes/5/Precomputed/WithAttributeSet-24 69.73n ± 16% 48.85n ± 14% -29.94% (p=0.002 n=6) EndToEndCounterAdd/NoFilter/Attributes/5/Precomputed/WithAttributes-24 61.32n ± 3% 49.01n ± 5% -20.08% (p=0.002 n=6) EndToEndCounterAdd/NoFilter/Attributes/5/Dynamic/WithAttributeSet-24 988.4n ± 48% 759.9n ± 55% -23.12% (p=0.041 n=6) EndToEndCounterAdd/NoFilter/Attributes/5/Dynamic/WithAttributes-24 954.1n ± 90% 667.9n ± 19% -30.00% (p=0.002 n=6) EndToEndCounterAdd/NoFilter/Attributes/5/Naive/WithAttributes-24 1.515µ ± 32% 1.233µ ± 12% ~ (p=0.240 n=6) EndToEndCounterAdd/NoFilter/Attributes/10/Precomputed/WithAttributeSet-24 70.86n ± 11% 53.90n ± 16% -23.94% (p=0.002 n=6) EndToEndCounterAdd/NoFilter/Attributes/10/Precomputed/WithAttributes-24 69.20n ± 21% 48.95n ± 12% -29.27% (p=0.004 n=6) EndToEndCounterAdd/NoFilter/Attributes/10/Dynamic/WithAttributeSet-24 1.483µ ± 28% 1.283µ ± 29% ~ (p=0.310 n=6) EndToEndCounterAdd/NoFilter/Attributes/10/Dynamic/WithAttributes-24 1.421µ ± 41% 1.664µ ± 49% ~ (p=0.394 n=6) EndToEndCounterAdd/NoFilter/Attributes/10/Naive/WithAttributes-24 2.784µ ± 66% 2.109µ ± 19% -24.23% (p=0.041 n=6) geomean 299.2n 252.1n -15.75% ``` --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…8286) Fixes #8285 Asynchronous instruments do not accept context, so the default TraceBased exemplar filter can never record an exemplar. Use the DropReservoir when the TraceBased exemplar filter is used, similar to #8211. ### Benchmarks Since callbacks are made as part of collect, there is a significant baseline overhead from collection included in the benchmark. ``` goos: linux goarch: amd64 pkg: go.opentelemetry.io/otel/sdk/metric cpu: AMD EPYC 7B12 │ main.txt │ new.txt │ │ sec/op │ sec/op vs base │ AsyncMeasureNewAttributeSet/AlwaysOn-24 3.662µ ± 13% 3.735µ ± 8% ~ (p=0.699 n=6) AsyncMeasureNewAttributeSet/TraceBased-24 3.586µ ± 10% 1.195µ ± 5% -66.69% (p=0.002 n=6) geomean 3.623µ 2.112µ -41.70% │ main.txt │ new.txt │ │ B/op │ B/op vs base │ AsyncMeasureNewAttributeSet/AlwaysOn-24 3.344Ki ± 0% 3.343Ki ± 0% ~ (p=0.571 n=6) AsyncMeasureNewAttributeSet/TraceBased-24 3424.5 ± 0% 592.0 ± 0% -82.71% (p=0.002 n=6) geomean 3.344Ki 1.390Ki -58.43% │ main.txt │ new.txt │ │ allocs/op │ allocs/op vs base │ AsyncMeasureNewAttributeSet/AlwaysOn-24 16.00 ± 0% 16.00 ± 0% ~ (p=1.000 n=6) ¹ AsyncMeasureNewAttributeSet/TraceBased-24 16.00 ± 0% 12.00 ± 0% -25.00% (p=0.002 n=6) geomean 16.00 13.86 -13.40% ¹ all samples are equal ```

### Added - Add `ByteSlice` and `ByteSliceValue` functions for new `BYTESLICE` attribute type in `go.opentelemetry.io/otel/attribute`. (#7948) - Apply attribute value limit to the `KindBytes` attribute type in `go.opentelemetry.io/otel/sdk/log`. (#7990) - Apply attribute value limit to the `BYTESLICE` attribute type in `go.opentelemetry.io/otel/sdk/trace`. (#7990) - Support `BYTESLICE` attributes in `go.opentelemetry.io/otel/trace`. (#8153) - Support `BYTESLICE` attributes in `go.opentelemetry.io/otel/exporters/otlp/otlptrace`. (#8153) - Support `BYTESLICE` attributes in `go.opentelemetry.io/otel/exporters/otlp/otlplog`. (#8153) - Support `BYTESLICE` attributes in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric`. (#8153) - Support `BYTESLICE` attributes in `go.opentelemetry.io/otel/exporters/zipkin`. (#8153) - Add `String` method for `Value` type in `go.opentelemetry.io/otel/attribute`. (#8142) - Add `Slice` and `SliceValue` functions for new `SLICE` attribute type in `go.opentelemetry.io/otel/attribute`. (#8166) - Support `SLICE` attributes in `go.opentelemetry.io/otel/exporters/otlp/otlptrace`. (#8216) - Support `SLICE` attributes in `go.opentelemetry.io/otel/exporters/otlp/otlplog`. (#8216) - Support `SLICE` attributes in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric`. (#8216) - Support `SLICE` attributes in `go.opentelemetry.io/otel/exporters/zipkin`. (#8216) - Apply `AttributeValueLengthLimit` to `attribute.SLICE` type attribute values in `go.opentelemetry.io/otel/sdk/trace`, recursively truncating contained string values. (#8217) - Add `Error` field on `Record` type in `go.opentelemetry.io/otel/log/logtest`. (#8148) - Add `WithMaxRequestSize` option in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`. (#8157) - Add `WithMaxRequestSize` option in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`. (#8157) - Add `WithMaxRequestSize` option in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`. (#8157) - Add `WithMaxRequestSize` option in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`. (#8157) - Add `WithMaxRequestSize` option in `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`. (#8157) - Add `WithMaxRequestSize` option in `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8157) - Add `Settable` to `go.opentelemetry.io/otel/metric/x` to allow reusing attribute options. (#8178) - Add experimental support for splitting metric data across multiple batches in `go.opentelemetry.io/otel/sdk/metric`. Set `OTEL_GO_X_METRIC_EXPORT_BATCH_SIZE=<max_size>` to enable for all periodic readers. See `go.opentelemetry.io/otel/sdk/metric/internal/x` for feature documentation. (#8071) - Add experimental self-observability metrics in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`. Enable with `OTEL_GO_X_SELF_OBSERVABILITY=true` environment variable. See `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc/internal/x` for feature documentation. (#8192) - Add experimental self-observability metrics in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`. Enable with `OTEL_GO_X_SELF_OBSERVABILITY=true` environment variable. See `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp/internal/x` for feature documentation. (#8194) - Add experimental self-observability metrics in `go.opentelemetry.io/otel/exporters/stdout/stdoutlog`. Enable with `OTEL_GO_X_SELF_OBSERVABILITY=true` environment variable. See `go.opentelemetry.io/otel/stdout/stdoutlog/internal/x` for feature documentation. (#8263) - Add `WithDefaultAttributes` to `go.opentelemetry.io/otel/metric/x` to support setting default attributes on instruments. (#8135) - Add `go.opentelemetry.io/otel/semconv/v1.41.0` package. The package contains semantic conventions from the `v1.41.0` version of the OpenTelemetry Semantic Conventions. See the [migration documentation](./semconv/v1.41.0/MIGRATION.md) for information on how to upgrade from `go.opentelemetry.io/otel/semconv/v1.40.0`. (#8324) - Add Observable variants of instruments to `go.opentelemetry.io/otel/semconv/v1.41.0` package. (#8350) - Generate explicit histogram bucket boundaries from weaver configuration for HTTP and RPC duration instruments in `go.opentelemetry.io/otel/semconv/v1.41.0`. (#8002) ### Changed - ⚠️ **Breaking Change:** `go.opentelemetry.io/otel/sdk/metric` now applies a default cardinality limit of 2000 to comply with the Metrics SDK specification recommendation. New attribute sets are dropped when the cardinality limit is reached. The measurement of these sets are aggregated into a special attribute set containing `attribute.Bool("otel.metric.overflow", true)`. This can break users who relied on the previous unlimited default. Set `WithCardinalityLimit(0)` or the deprecated `OTEL_GO_X_CARDINALITY_LIMIT=0` environment variable to preserve unlimited cardinality. Note that support for `OTEL_GO_X_CARDINALITY_LIMIT` may be removed in a future release. (#8247) - `ErrorType` in `go.opentelemetry.io/otel/semconv` now unwraps errors created with `fmt.Errorf` when deriving the `error.type` attribute. (#8133) - `go.opentelemetry.io/otel/sdk/log` now unwraps error chains created with `fmt.Errorf` when deriving the `error.type` attribute from errors on log records. (#8133) - `Set.MarshalLog` method in `go.opentelemetry.io/otel/attribute` now uses `Value.String` formatting following the [OpenTelemetry AnyValue representation for non-OTLP protocols](https://opentelemetry.io/docs/specs/otel/common/#anyvalue). (#8169) - Optimize `go.opentelemetry.io/otel/sdk/metric` to return a drop reservoir and short-circuit `Offer` calls to the exemplar reservoir when `exemplar.AlwaysOffFilter` is configured. (#8211) (#8267) - Optimize `go.opentelemetry.io/otel/sdk/metric` to return a drop reservoir for asynchronous instruments when `exemplar.TraceBasedFilter` is configured. (#8286) ### Deprecated - Deprecate `Value.Emit` method in `go.opentelemetry.io/otel/attribute`. Use `Value.String` instead. (#8176) ### Fixed - Limit OTLP request size to 64 MiB by default in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`. The limit applies before compression, oversized requests are treated as non-retryable errors, and the limit can be configured with the new `WithMaxRequestSize` option. (#8157, #8365) - Limit OTLP request size to 64 MiB by default in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`. The limit applies before compression, oversized requests are treated as non-retryable errors, and the limit can be configured with the new `WithMaxRequestSize` option. (#8157, #8365) - Limit OTLP request size to 64 MiB by default in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`. The limit applies before compression, oversized requests are treated as non-retryable errors, and the limit can be configured with the new `WithMaxRequestSize` option. (#8157, #8365) - Limit OTLP request size to 64 MiB by default in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`. The limit applies before compression, oversized requests are treated as non-retryable errors, and the limit can be configured with the new `WithMaxRequestSize` option. (#8157, #8365) - Limit OTLP request size to 64 MiB by default in `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`. The limit applies before compression, oversized requests are treated as non-retryable errors, and the limit can be configured with the new `WithMaxRequestSize` option. (#8157, #8365) - Limit OTLP request size to 64 MiB by default in `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. The limit applies before compression, oversized requests are treated as non-retryable errors, and the limit can be configured with the new `WithMaxRequestSize` option. (#8157, #8365) - Fix gzipped request body replay on redirect in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`. (#8135) - Fix gzipped request body replay on redirect in `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8152) - `go.opentelemetry.io/otel/exporters/prometheus` now uses `Value.String` formatting for label values following the [OpenTelemetry AnyValue representation for non-OTLP protocols](https://opentelemetry.io/docs/specs/otel/common/#anyvalue). (#8170) - Propagate errors from the exporter when calling `Shutdown` on `BatchSpanProcessor` in `go.opentelemetry.io/otel/sdk/trace`. (#8197) - Fix stale status code reporting on self-observability metrics in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` and `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8226) - Fix a concurrent `Collect` data race and potential panic in `go.opentelemetry.io/otel/exporters/prometheus` when `WithResourceAsConstantLabels` option is used. (#8227) - Fix race condition in `FixedSizeReservoir` in `go.opentelemetry.io/otel/sdk/metric/exemplar` by reverting #7447. (#8249) - Fix `FixedSizeReservoir` in `go.opentelemetry.io/otel/sdk/metric/exemplar` to safely handle zero size. A capacity check in the constructor initializes the reservoir safely and skips initialization for zero-cap; early returns in `Offer()` and `Collect()` ensure no-op behavior. (#8295) - Fix counting of spans and logs in self-observability metrics in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`, `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`, `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`, and `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8254) - Drop conflicting scope attributes named `name`, `version`, or `schema_url` from metric labels in `go.opentelemetry.io/otel/exporters/prometheus`, preserving the dedicated `otel_scope_name`, `otel_scope_version`, and `otel_scope_schema_url` labels. (#8264) - Close schema files opened by `ParseFile` in `go.opentelemetry.io/otel/schema/v1.0` and `go.opentelemetry.io/otel/schema/v1.1`. ([GHSA-995v-fvrw-c78m](GHSA-995v-fvrw-c78m)) - Enforce the 8192-byte baggage size limit during extraction/parsing, changing behavior when the limit is exceeded in `go.opentelemetry.io/otel/baggage` and `go.opentelemetry.io/otel/propagation`. (#8222) - Fix `go.opentelemetry.io/otel/semconv/v1.41.0` to include `Attr*` helper methods for required attributes on observable instruments. (#8361) - Limit baggage extraction error reporting in `go.opentelemetry.io/otel/propagation` to prevent malformed or oversized baggage headers from flooding logs. ([GHSA-5wrp-cwcj-q835](GHSA-5wrp-cwcj-q835))

dashpole marked this pull request as ready for review April 15, 2026 15:55

dashpole requested review from MrAlias, XSAM, dmathieu, flc1125 and pellared as code owners April 15, 2026 15:55

dashpole requested a review from Copilot April 15, 2026 18:40

Copilot started reviewing on behalf of dashpole April 15, 2026 18:40 View session

Copilot AI reviewed Apr 15, 2026

View reviewed changes

Comment thread CHANGELOG.md Outdated

Comment thread sdk/metric/exemplar.go

Comment thread sdk/metric/exemplar_test.go Outdated

Comment thread sdk/metric/internal/aggregate/aggregate.go

dashpole force-pushed the cheap_alwaysoff branch from 49d0537 to 2463fed Compare April 16, 2026 01:47

dashpole requested a review from Copilot April 16, 2026 14:13

Copilot started reviewing on behalf of dashpole April 16, 2026 14:13 View session

Copilot AI reviewed Apr 16, 2026

View reviewed changes

Comment thread sdk/metric/exemplar.go Outdated

Comment thread CHANGELOG.md Outdated

dashpole force-pushed the cheap_alwaysoff branch from 53bc0e1 to 01baf21 Compare April 16, 2026 16:54

dashpole requested a review from Copilot April 16, 2026 16:55

Copilot started reviewing on behalf of dashpole April 16, 2026 16:55 View session

Copilot AI reviewed Apr 16, 2026

View reviewed changes

Comment thread sdk/metric/config.go Outdated

Comment thread sdk/metric/benchmark_test.go

dashpole and others added 6 commits April 16, 2026 18:38

use a DropReservoir when an exemplar.AlwaysOffFilter is provided

059e92b

Apply suggestions from code review

6de2155

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

avoid using reflection in the test

714c327

update comment

12a11af

address comments

25e8244

make sure benchmark doesn't use too much memory

f252a8d

dashpole force-pushed the cheap_alwaysoff branch from 01baf21 to f252a8d Compare April 16, 2026 18:38

dashpole requested a review from Copilot April 16, 2026 18:42

dashpole mentioned this pull request Apr 16, 2026

Fix typos found by copilot #8221

Merged

Copilot started reviewing on behalf of dashpole April 16, 2026 18:43 View session

Copilot AI reviewed Apr 16, 2026

View reviewed changes

MrAlias pushed a commit that referenced this pull request Apr 16, 2026

Fix typos found by copilot (#8221)

3356ad3

Found during copilot code reviews on other PRs. #8211 (comment)

MrAlias approved these changes Apr 16, 2026

View reviewed changes

MrAlias reviewed Apr 16, 2026

View reviewed changes

Comment thread sdk/metric/exemplar.go Outdated

pellared reviewed Apr 20, 2026

View reviewed changes

Comment thread CHANGELOG.md Outdated

Comment thread sdk/metric/config.go Outdated

don't handle nil exemplar filter, and update changelog

14dad02

dashpole force-pushed the cheap_alwaysoff branch from be389a0 to 14dad02 Compare April 20, 2026 20:30

dashpole mentioned this pull request Apr 20, 2026

Decide how to handle metric.WithExemplarFilter(nil) #8240

Closed

Merge branch 'main' into cheap_alwaysoff

bf29be0

dashpole requested a review from pellared April 21, 2026 13:12

Merge branch 'main' into cheap_alwaysoff

76b8753

dmathieu approved these changes Apr 21, 2026

View reviewed changes

flc1125 reviewed Apr 21, 2026

View reviewed changes

Comment thread sdk/metric/internal/aggregate/drop.go

dashpole added 2 commits April 21, 2026 14:59

Merge branch 'main' into cheap_alwaysoff

184fff3

Merge branch 'main' into cheap_alwaysoff

dda4983

pellared approved these changes Apr 21, 2026

View reviewed changes

dashpole merged commit 6f81801 into open-telemetry:main Apr 21, 2026
34 checks passed

dashpole deleted the cheap_alwaysoff branch April 21, 2026 20:08

pellared pushed a commit to pellared/opentelemetry-go that referenced this pull request Apr 23, 2026

Fix typos found by copilot (open-telemetry#8221)

946e24d

Found during copilot code reviews on other PRs. open-telemetry#8211 (comment)

This was referenced Apr 27, 2026

Optimize metrics sdk measurement with AlwaysOff exemplar filter #8267

Merged

Disable exemplar reservoirs for asynchronous instruments with a TraceBased exemplar filter #8285

Closed

Disable exemplar reservoir for asynchronous instruments by default #8286

Merged

pellared mentioned this pull request May 26, 2026

Release 1.44.0 #8376

Merged

pellared added this to the v1.44.0 milestone May 27, 2026

kln21002 mentioned this pull request Jun 10, 2026

sdk/metrics: Excessive Memory usage in Metrics Reader (1.33.0) #6333

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use a DropReservoir when an exemplar.AlwaysOffFilter is provided#8211

Use a DropReservoir when an exemplar.AlwaysOffFilter is provided#8211
dashpole merged 11 commits into
open-telemetry:mainfrom
dashpole:cheap_alwaysoff

dashpole commented Apr 15, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

dashpole commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

codecov Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dashpole commented Apr 15, 2026 •

edited

Loading

codecov Bot commented Apr 15, 2026 •

edited

Loading