Skip to content

feat: add self-observability metrics to otlpmetricgrpc metric exporters#8192

Merged
pellared merged 48 commits into
open-telemetry:mainfrom
dashpole:pr-7120-new
May 27, 2026
Merged

feat: add self-observability metrics to otlpmetricgrpc metric exporters#8192
pellared merged 48 commits into
open-telemetry:mainfrom
dashpole:pr-7120-new

Conversation

@dashpole

@dashpole dashpole commented Apr 13, 2026

Copy link
Copy Markdown
Collaborator

Fixes #7012

This updates #7120, and addresses outstanding comments. See commit descriptions for the changes made on top of that PR. Credit to @minimAluminiumalism for the initial work.

Benchmark results:

goos: linux
goarch: amd64
pkg: go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ
cpu: Intel(R) Xeon(R) CPU @ 2.20GHz
                                             │ bench_feature.txt │
                                             │      sec/op       │
InstrumentationTrackExport/EnabledNoError-24         1.045µ ± 5%
InstrumentationTrackExport/EnabledError-24           2.248µ ± 1%
InstrumentationTrackExport/Disabled-24               9.643n ± 1%
geomean                                              282.9n

                                             │ bench_feature.txt │
                                             │       B/op        │
InstrumentationTrackExport/EnabledNoError-24        160.0 ± 0%
InstrumentationTrackExport/EnabledError-24          488.0 ± 0%
InstrumentationTrackExport/Disabled-24              0.000 ± 0%
geomean                                                        ¹
¹ summaries must be >0 to compute geomean

                                             │ bench_feature.txt │
                                             │     allocs/op     │
InstrumentationTrackExport/EnabledNoError-24        5.000 ± 0%
InstrumentationTrackExport/EnabledError-24          6.000 ± 0%
InstrumentationTrackExport/Disabled-24              0.000 ± 0%
geomean                                                        ¹
¹ summaries must be >0 to compute geomean

Written with assistance from Gemini.

@codecov

codecov Bot commented Apr 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.53112% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.0%. Comparing base (761bbfc) to head (ebb15ec).

Files with missing lines Patch % Lines
.../otlpmetricgrpc/internal/observ/instrumentation.go 90.0% 11 Missing and 5 partials ⚠️
...porters/otlp/otlpmetric/otlpmetricgrpc/exporter.go 87.5% 1 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##            main   #8192     +/-   ##
=======================================
+ Coverage   82.9%   83.0%   +0.1%     
=======================================
  Files        315     320      +5     
  Lines      25115   25354    +239     
=======================================
+ Hits       20835   21059    +224     
- Misses      3909    3917      +8     
- Partials     371     378      +7     
Files with missing lines Coverage Δ
...pmetric/otlpmetricgrpc/internal/counter/counter.go 100.0% <100.0%> (ø)
...tlpmetric/otlpmetricgrpc/internal/observ/target.go 100.0% <100.0%> (ø)
...tlp/otlpmetric/otlpmetricgrpc/internal/x/observ.go 100.0% <100.0%> (ø)
...ers/otlp/otlpmetric/otlpmetricgrpc/internal/x/x.go 100.0% <100.0%> (ø)
...porters/otlp/otlpmetric/otlpmetricgrpc/exporter.go 84.2% <87.5%> (+0.3%) ⬆️
.../otlpmetricgrpc/internal/observ/instrumentation.go 90.0% <90.0%> (ø)

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dashpole dashpole marked this pull request as ready for review April 14, 2026 03:44
Comment thread CHANGELOG.md Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds experimental self-observability metrics to the OTLP gRPC metric exporter (otlpmetricgrpc), gated behind OTEL_GO_X_OBSERVABILITY, aligning with the SDK metrics self-observability semantic conventions.

Changes:

  • Introduces an internal/observ instrumentation package that records inflight/exported datapoints and export duration for otlpmetricgrpc.
  • Adds an internal/x feature-flag package (and docs) to gate the experimental behavior via OTEL_GO_X_OBSERVABILITY.
  • Adds generated internal/counter for deterministic component instance IDs plus new tests/benchmarks and changelog entry.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
exporters/otlp/otlpmetric/otlpmetricgrpc/exporter.go Hooks exporter Export into the new self-observability instrumentation.
exporters/otlp/otlpmetric/otlpmetricgrpc/go.mod Promotes go.opentelemetry.io/otel/metric to a direct dependency for instrumentation usage.
exporters/otlp/otlpmetric/otlpmetricgrpc/doc.go Links exporter docs to experimental feature documentation.
exporters/otlp/otlpmetric/otlpmetricgrpc/selfobservability_test.go End-to-end tests validating exporter self-observability emission (enabled/disabled/error).
exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation.go Implements the three self-observability metrics and endpoint parsing.
exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation_test.go Unit tests + benchmark coverage for the instrumentation behavior.
exporters/otlp/otlpmetric/otlpmetricgrpc/internal/x/x.go Implements the OTEL_GO_X_OBSERVABILITY feature flag parsing.
exporters/otlp/otlpmetric/otlpmetricgrpc/internal/x/x_test.go Validates feature flag parsing/enablement behavior.
exporters/otlp/otlpmetric/otlpmetricgrpc/internal/x/README.md Documents experimental features and self-observability semantics.
exporters/otlp/otlpmetric/otlpmetricgrpc/internal/gen.go Adds go:generate for the shared counter template.
exporters/otlp/otlpmetric/otlpmetricgrpc/internal/counter/counter.go Generated atomic exporter ID counter for unique component names.
exporters/otlp/otlpmetric/otlpmetricgrpc/internal/counter/counter_test.go Tests counter reset/increment + concurrency safety.
CHANGELOG.md Notes the new experimental self-observability feature.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation_test.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/selfobservability_test.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/selfobservability_test.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/selfobservability_test.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Comment thread CHANGELOG.md Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/exporter.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/exporter.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/selfobservability_test.go
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/selfobservability_test.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation_test.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation_test.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/selfobservability_test.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/x/x_test.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation_test.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/exporter.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/internal/observ/instrumentation.go Outdated
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/exporter.go
Comment thread exporters/otlp/otlpmetric/otlpmetricgrpc/exporter.go Outdated
dashpole and others added 24 commits May 27, 2026 13:51
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: David Ashpole <dashpole@google.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…umentation_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@pellared pellared merged commit 6fdcf82 into open-telemetry:main May 27, 2026
33 of 34 checks passed
@pellared pellared mentioned this pull request May 27, 2026
pellared added a commit that referenced this pull request May 27, 2026
### Added

- Add `ByteSlice` and `ByteSliceValue` functions for new `BYTESLICE`
attribute type in `go.opentelemetry.io/otel/attribute`. (#7948)
- Apply attribute value limit to the `KindBytes` attribute type in
`go.opentelemetry.io/otel/sdk/log`. (#7990)
- Apply attribute value limit to the `BYTESLICE` attribute type in
`go.opentelemetry.io/otel/sdk/trace`. (#7990)
- Support `BYTESLICE` attributes in `go.opentelemetry.io/otel/trace`.
(#8153)
- Support `BYTESLICE` attributes in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace`. (#8153)
- Support `BYTESLICE` attributes in
`go.opentelemetry.io/otel/exporters/otlp/otlplog`. (#8153)
- Support `BYTESLICE` attributes in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric`. (#8153)
- Support `BYTESLICE` attributes in
`go.opentelemetry.io/otel/exporters/zipkin`. (#8153)
- Add `String` method for `Value` type in
`go.opentelemetry.io/otel/attribute`. (#8142)
- Add `Slice` and `SliceValue` functions for new `SLICE` attribute type
in `go.opentelemetry.io/otel/attribute`. (#8166)
- Support `SLICE` attributes in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace`. (#8216)
- Support `SLICE` attributes in
`go.opentelemetry.io/otel/exporters/otlp/otlplog`. (#8216)
- Support `SLICE` attributes in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric`. (#8216)
- Support `SLICE` attributes in
`go.opentelemetry.io/otel/exporters/zipkin`. (#8216)
- Apply `AttributeValueLengthLimit` to `attribute.SLICE` type attribute
values in `go.opentelemetry.io/otel/sdk/trace`, recursively truncating
contained string values. (#8217)
- Add `Error` field on `Record` type in
`go.opentelemetry.io/otel/log/logtest`. (#8148)
- Add `WithMaxRequestSize` option in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`.
(#8157)
- Add `WithMaxRequestSize` option in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`.
(#8157)
- Add `WithMaxRequestSize` option in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`.
(#8157)
- Add `WithMaxRequestSize` option in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`.
(#8157)
- Add `WithMaxRequestSize` option in
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`. (#8157)
- Add `WithMaxRequestSize` option in
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8157)
- Add `Settable` to `go.opentelemetry.io/otel/metric/x` to allow reusing
attribute options. (#8178)
- Add experimental support for splitting metric data across multiple
batches in `go.opentelemetry.io/otel/sdk/metric`.
Set `OTEL_GO_X_METRIC_EXPORT_BATCH_SIZE=<max_size>` to enable for all
periodic readers.
See `go.opentelemetry.io/otel/sdk/metric/internal/x` for feature
documentation. (#8071)
- Add experimental self-observability metrics in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`.
  Enable with `OTEL_GO_X_SELF_OBSERVABILITY=true` environment variable.
See
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc/internal/x`
for feature documentation. (#8192)
- Add experimental self-observability metrics in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`.
  Enable with `OTEL_GO_X_SELF_OBSERVABILITY=true` environment variable.
See
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp/internal/x`
for feature documentation. (#8194)
- Add experimental self-observability metrics in
`go.opentelemetry.io/otel/exporters/stdout/stdoutlog`.
  Enable with `OTEL_GO_X_SELF_OBSERVABILITY=true` environment variable.
See `go.opentelemetry.io/otel/stdout/stdoutlog/internal/x` for feature
documentation. (#8263)
- Add `WithDefaultAttributes` to `go.opentelemetry.io/otel/metric/x` to
support setting default attributes on instruments. (#8135)
- Add `go.opentelemetry.io/otel/semconv/v1.41.0` package.
The package contains semantic conventions from the `v1.41.0` version of
the OpenTelemetry Semantic Conventions.
See the [migration documentation](./semconv/v1.41.0/MIGRATION.md) for
information on how to upgrade from
`go.opentelemetry.io/otel/semconv/v1.40.0`. (#8324)
- Add Observable variants of instruments to
`go.opentelemetry.io/otel/semconv/v1.41.0` package. (#8350)
- Generate explicit histogram bucket boundaries from weaver
configuration for HTTP and RPC duration instruments in
`go.opentelemetry.io/otel/semconv/v1.41.0`. (#8002)

### Changed

- ⚠️ **Breaking Change:** `go.opentelemetry.io/otel/sdk/metric` now
applies a default cardinality limit of 2000 to comply with the Metrics
SDK specification recommendation.
New attribute sets are dropped when the cardinality limit is reached.
The measurement of these sets are aggregated into a special attribute
set containing `attribute.Bool("otel.metric.overflow", true)`.
  This can break users who relied on the previous unlimited default.
Set `WithCardinalityLimit(0)` or the deprecated
`OTEL_GO_X_CARDINALITY_LIMIT=0` environment variable to preserve
unlimited cardinality.
Note that support for `OTEL_GO_X_CARDINALITY_LIMIT` may be removed in a
future release. (#8247)
- `ErrorType` in `go.opentelemetry.io/otel/semconv` now unwraps errors
created with `fmt.Errorf` when deriving the `error.type` attribute.
(#8133)
- `go.opentelemetry.io/otel/sdk/log` now unwraps error chains created
with `fmt.Errorf` when deriving the `error.type` attribute from errors
on log records. (#8133)
- `Set.MarshalLog` method in `go.opentelemetry.io/otel/attribute` now
uses `Value.String` formatting following the [OpenTelemetry AnyValue
representation for non-OTLP
protocols](https://opentelemetry.io/docs/specs/otel/common/#anyvalue).
(#8169)
- Optimize `go.opentelemetry.io/otel/sdk/metric` to return a drop
reservoir and short-circuit `Offer` calls to the exemplar reservoir when
`exemplar.AlwaysOffFilter` is configured. (#8211) (#8267)
- Optimize `go.opentelemetry.io/otel/sdk/metric` to return a drop
reservoir for asynchronous instruments when `exemplar.TraceBasedFilter`
is configured. (#8286)

### Deprecated

- Deprecate `Value.Emit` method in `go.opentelemetry.io/otel/attribute`.
  Use `Value.String` instead. (#8176)

### Fixed

- Limit OTLP request size to 64 MiB by default in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`.
The limit applies before compression, oversized requests are treated as
non-retryable errors, and the limit can be configured with the new
`WithMaxRequestSize` option. (#8157, #8365)
- Limit OTLP request size to 64 MiB by default in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`.
The limit applies before compression, oversized requests are treated as
non-retryable errors, and the limit can be configured with the new
`WithMaxRequestSize` option. (#8157, #8365)
- Limit OTLP request size to 64 MiB by default in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`.
The limit applies before compression, oversized requests are treated as
non-retryable errors, and the limit can be configured with the new
`WithMaxRequestSize` option. (#8157, #8365)
- Limit OTLP request size to 64 MiB by default in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`.
The limit applies before compression, oversized requests are treated as
non-retryable errors, and the limit can be configured with the new
`WithMaxRequestSize` option. (#8157, #8365)
- Limit OTLP request size to 64 MiB by default in
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`.
The limit applies before compression, oversized requests are treated as
non-retryable errors, and the limit can be configured with the new
`WithMaxRequestSize` option. (#8157, #8365)
- Limit OTLP request size to 64 MiB by default in
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`.
The limit applies before compression, oversized requests are treated as
non-retryable errors, and the limit can be configured with the new
`WithMaxRequestSize` option. (#8157, #8365)
- Fix gzipped request body replay on redirect in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`.
(#8135)
- Fix gzipped request body replay on redirect in
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8152)
- `go.opentelemetry.io/otel/exporters/prometheus` now uses
`Value.String` formatting for label values following the [OpenTelemetry
AnyValue representation for non-OTLP
protocols](https://opentelemetry.io/docs/specs/otel/common/#anyvalue).
(#8170)
- Propagate errors from the exporter when calling `Shutdown` on
`BatchSpanProcessor` in `go.opentelemetry.io/otel/sdk/trace`. (#8197)
- Fix stale status code reporting on self-observability metrics in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` and
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8226)
- Fix a concurrent `Collect` data race and potential panic in
`go.opentelemetry.io/otel/exporters/prometheus` when
`WithResourceAsConstantLabels` option is used. (#8227)
- Fix race condition in `FixedSizeReservoir` in
`go.opentelemetry.io/otel/sdk/metric/exemplar` by reverting #7447.
(#8249)
- Fix `FixedSizeReservoir` in
`go.opentelemetry.io/otel/sdk/metric/exemplar` to safely handle zero
size.
A capacity check in the constructor initializes the reservoir safely and
skips initialization for zero-cap; early returns in `Offer()` and
`Collect()` ensure no-op behavior. (#8295)
- Fix counting of spans and logs in self-observability metrics in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`,
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`,
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`, and
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8254)
- Drop conflicting scope attributes named `name`, `version`, or
`schema_url` from metric labels in
`go.opentelemetry.io/otel/exporters/prometheus`, preserving the
dedicated `otel_scope_name`, `otel_scope_version`, and
`otel_scope_schema_url` labels. (#8264)
- Close schema files opened by `ParseFile` in
`go.opentelemetry.io/otel/schema/v1.0` and
`go.opentelemetry.io/otel/schema/v1.1`.
([GHSA-995v-fvrw-c78m](GHSA-995v-fvrw-c78m))
- Enforce the 8192-byte baggage size limit during extraction/parsing,
changing behavior when the limit is exceeded in
`go.opentelemetry.io/otel/baggage` and
`go.opentelemetry.io/otel/propagation`. (#8222)
- Fix `go.opentelemetry.io/otel/semconv/v1.41.0` to include `Attr*`
helper methods for required attributes on observable instruments.
(#8361)
- Limit baggage extraction error reporting in
`go.opentelemetry.io/otel/propagation` to prevent malformed or oversized
baggage headers from flooding logs.
([GHSA-5wrp-cwcj-q835](GHSA-5wrp-cwcj-q835))
@pellared pellared added this to the v1.44.0 milestone May 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metrics SDK observability - otlpmetricgrpc exporter metrics

6 participants