Add support for consuming OTLP/gRPC metrics by axw · Pull Request #4722 · elastic/apm-server

axw · 2021-02-11T05:19:05Z

Motivation/summary

Add support for consuming basic OTLP/gRPC metrics,
like we support in the OpenTelemetry Collector exporter.

We do not yet support distributions or summaries; these
will be added once the Elasticsearch enhancements we
are dependent on are available. If these are received, the
server will ignore them and increment a monitoring counter
to indicate that an unsupported metric was received and
dropped.

Checklist

Update CHANGELOG.asciidoc
~~- [ ] Documentation has been updated~~ [OTLP] Document support for OTLP/gRPC on the standard port #4678

How to test these changes

Start apm-server with monitoring enabled
Run an application that is instrumented with OpenTelemetry to collect metrics (including some distributions and summaries), and configured to export OTLP to APM Server's standard host:port.
Repeat step 2, exporting to an OpenTelemetry Collector with the elastic exporter.
Compare the data received directly by APM Server to the data sent by the exporter.
Check the beats monitoring doc for apm-server, and check the value for apm-server.otlp.grpc.metrics.consumer.unsupported_dropped corresponds to the number of distribution and summary metrics sent

Related issues

Closes #4503

ghost · 2021-02-11T05:22:58Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #4722 updated
Start Time: 2021-02-16T01:42:34.255+0000
Duration: 56 min 20 sec
Commit: 53a9a74

Test stats 🧪

Test	Results
Failed	0
Passed	4733
Skipped	124
Total	4857

Trends 🧪

Steps errors

Expand to view the steps failures

`Run Window tests`

Took 10 min 6 sec . View more details on here

`Compress`

Took 0 min 0 sec . View more details on here
Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

`Compress`

Took 0 min 0 sec . View more details on here
Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

`Test Sync`

Took 3 min 26 sec . View more details on here
Description: ./.ci/scripts/sync.sh

Add support for consuming basic OTLP/gRPC metrics, like we support in the OpenTelemetry Collector exporter. This does not cover distributions or summaries; these will be added once the Elasticsearch enhancements we are dependent on are available.

codecov-io · 2021-02-11T07:49:38Z

Codecov Report

Merging #4722 (53a9a74) into master (a25b110) will increase coverage by 0.26%.
The diff coverage is 97.72%.

@@            Coverage Diff             @@
##           master    #4722      +/-   ##
==========================================
+ Coverage   76.43%   76.70%   +0.26%     
==========================================
  Files         165      166       +1     
  Lines       10083    10211     +128     
==========================================
+ Hits         7707     7832     +125     
- Misses       2376     2379       +3

Impacted Files	Coverage Δ
beater/otlp/grpc.go	`87.87% <91.30%> (+2.16%)`	⬆️
processor/otel/metrics.go	`99.06% <99.06%> (ø)`
processor/otel/consumer.go	`94.89% <100.00%> (+0.02%)`	⬆️

jalvz · 2021-02-11T11:07:00Z

processor/otel/consumer.go

+	UnsupportedMetricsDropped int64
+}
+
+type consumerStats struct {


I am confused about this, why both consumerStats and ConsumerStats are needed?

consumerStats might change at any time, ConsumerStats is a snapshot. I'll add a comment.

jalvz · 2021-02-11T11:11:34Z

processor/otel/metrics.go

+	otelMetrics := in.Metrics()
+	var unsupported int64
+	for i := 0; i < otelMetrics.Len(); i++ {
+		if !c.addMetric(otelMetrics.At(i), &ms) {


instead of having a method that both mutates and argument and returns a value, can't we say unsupported += otelMetrics.Len() - ms.Len()?

No, because two metrics may end up in the same metricset. (ms is a collection of metricsets)

jalvz · 2021-02-11T11:16:24Z

processor/otel/metrics.go

+func (c *Consumer) addMetric(metric pdata.Metric, ms *metricsets) bool {
+	switch metric.DataType() {
+	case pdata.MetricDataTypeIntGauge:
+		dps := metric.IntGauge().DataPoints()


Maybe dumb question, but can't we make all these data types conform an interface defining LabelsMap(), Timestamp() and Value()?
All these cases are the same

Yes, but:

That won't work when we add support for distributions and summaries

Ideally we would also support distinguishing doubles from ints, storing the latter in a long field type

I'd prefer to leave it as it is for now (this is also how it looks in opentelemetry-collector-contrib), and refactor it later if it's a pain.

axw · 2021-02-15T02:50:10Z

jenkins run the tests please

* Add support for consuming OTLP/gRPC metrics Add support for consuming basic OTLP/gRPC metrics, like we support in the OpenTelemetry Collector exporter. This does not cover distributions or summaries; these will be added once the Elasticsearch enhancements we are dependent on are available. * processor/otel: add comment about [Cc]onsumerStats # Conflicts: # changelogs/head.asciidoc

…chemas-to-agents * upstream/master: (111 commits) Introduce metricset.name (elastic#4857) processor/otel: test service.version handling (elastic#4853) docs: Add PHP agent information to shared docs (elastic#4740) Script for faster development workflow (elastic#4731) Update to elastic/beats@1b31c26 (elastic#4763) backport: add 7.12 to .backportrc.json (elastic#4807) backport: enable auto-merge on backport PRs (elastic#4777) Support for Node.js profiles (elastic#4728) docs: readds .NET link (elastic#4764) [DOCS] Fixes URLs on Secure communication with APM Agents page (elastic#4746) ci: set proper parameters for the tar step (elastic#4696) docs: add 7.11.1 release notes (elastic#4727) Disable sourcemap upload endpoint when data streams enabled (elastic#4735) Add service name to dataset field (elastic#4674) Update to elastic/beats@ba423212a660 (elastic#4733) sampling: require a default policy (elastic#4729) processor/otel: add unit test for span status (elastic#4734) Add support for consuming OTLP/gRPC metrics (elastic#4722) [apmpackage] Add config options supported in ESS (elastic#4690) Use the apm-server version everywhere* (elastic#4725) ...

…te-schema-json-1 * upstream/master: (111 commits) Introduce metricset.name (elastic#4857) processor/otel: test service.version handling (elastic#4853) docs: Add PHP agent information to shared docs (elastic#4740) Script for faster development workflow (elastic#4731) Update to elastic/beats@1b31c26 (elastic#4763) backport: add 7.12 to .backportrc.json (elastic#4807) backport: enable auto-merge on backport PRs (elastic#4777) Support for Node.js profiles (elastic#4728) docs: readds .NET link (elastic#4764) [DOCS] Fixes URLs on Secure communication with APM Agents page (elastic#4746) ci: set proper parameters for the tar step (elastic#4696) docs: add 7.11.1 release notes (elastic#4727) Disable sourcemap upload endpoint when data streams enabled (elastic#4735) Add service name to dataset field (elastic#4674) Update to elastic/beats@ba423212a660 (elastic#4733) sampling: require a default policy (elastic#4729) processor/otel: add unit test for span status (elastic#4734) Add support for consuming OTLP/gRPC metrics (elastic#4722) [apmpackage] Add config options supported in ESS (elastic#4690) Use the apm-server version everywhere* (elastic#4725) ...

simitt · 2021-03-04T13:17:38Z

Tested with BC 2; metrics are accepted and ingested as expected.

Regarding testing of distributions or summaries - how did you test that when implementing? I was instrumenting a go app, but the otel go metrics implementation only supports counters so far.

axw · 2021-03-05T01:57:41Z

@simitt it does support histograms and summaries. You need to define an aggregator in the metrics controller, and then use a ValueRecorder meter. Frankly speaking their metrics API is obtuse, so you can't be faulted for missing that :)

Here's how I tested histograms:

apm-server/systemtest/otlp_test.go

Lines 174 to 193 in d0624d4

    
           controller := controller.New( 
        
           	processor.New( 
        
           		simple.NewWithHistogramDistribution([]float64{1, 100, 1000, 10000}), 
        
           		exporter, 
        
           	), 
        
           	controller.WithPusher(exporter), 
        
           	controller.WithCollectPeriod(time.Minute), 
        
           ) 
        
           if err := controller.Start(context.Background()); err != nil { 
        
           	return err 
        
           } 
        
           meterProvider := controller.MeterProvider() 
        
           meter := meterProvider.Meter("test-meter") 
        
           float64Counter := metric.Must(meter).NewFloat64Counter("float64_counter") 
        
           float64Counter.Add(context.Background(), 1) 
        
           // This will be dropped, as we do not support consuming histograms yet. 
        
           int64Recorder := metric.Must(meter).NewInt64ValueRecorder("int64_recorder") 
        
           int64Recorder.Record(context.Background(), 123)

I believe you need to change NewWithHistogramDistribution to NewWithInexpensiveDistribution to aggregate the recordings as a summary instead.

simitt · 2021-03-05T08:52:30Z

I was missing that one could use simple.NewWithInexpensiveDistribution() - thanks.

Tested with BC3:

compared metrics for data sent via otel exporter and elastic exporter - no differences found
works as expected when using the otel exporter to directly send data to APM Server
when using the elastic exporter, no beats_stats.metrics.apm-server.otlp.grpc.metrics.consumer.unsupported_dropped are recorded (probably expected that this is not supported, but mentioning it due to the testing instructions above)
otlp data are not indexed, as they are missing in the template definition. This is probably fine as no aggregations are currently done on the data.

axw added test-plan v7.12.0 labels Feb 11, 2021

axw marked this pull request as draft February 11, 2021 05:19

axw force-pushed the otel-metrics branch from e0fbf7b to ec30095 Compare February 11, 2021 05:21

axw force-pushed the otel-metrics branch from ec30095 to 4da9e41 Compare February 11, 2021 07:29

axw marked this pull request as ready for review February 11, 2021 08:50

axw requested a review from a team February 11, 2021 08:50

Merge branch 'master' into otel-metrics

8cd5920

jalvz reviewed Feb 11, 2021

View reviewed changes

processor/otel: add comment about [Cc]onsumerStats

fd54544

axw requested a review from jalvz February 11, 2021 12:57

jalvz approved these changes Feb 15, 2021

View reviewed changes

Merge branch 'master' into otel-metrics

53a9a74

axw merged commit 6d8ad81 into elastic:master Feb 16, 2021

axw deleted the otel-metrics branch February 16, 2021 02:56

axw mentioned this pull request Feb 19, 2021

[7.x] Add support for consuming OTLP/gRPC metrics (#4722) #4825

Merged

axw mentioned this pull request Feb 19, 2021

[7.12] Add support for consuming OTLP/gRPC metrics (#4722) #4826

Merged

simitt self-assigned this Mar 2, 2021

Conversation

axw commented Feb 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation/summary

Checklist

How to test these changes

Related issues

Uh oh!

ghost commented Feb 11, 2021 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

Steps errors

Run Window tests

Compress

Compress

Test Sync

Uh oh!

codecov-io commented Feb 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jalvz Feb 11, 2021

Choose a reason for hiding this comment

Uh oh!

axw Feb 11, 2021

Choose a reason for hiding this comment

Uh oh!

jalvz Feb 11, 2021

Choose a reason for hiding this comment

Uh oh!

axw Feb 11, 2021

Choose a reason for hiding this comment

Uh oh!

jalvz Feb 11, 2021

Choose a reason for hiding this comment

Uh oh!

axw Feb 11, 2021

Choose a reason for hiding this comment

Uh oh!

axw commented Feb 15, 2021

Uh oh!

simitt commented Mar 4, 2021

Uh oh!

axw commented Mar 5, 2021

Uh oh!

simitt commented Mar 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

axw commented Feb 11, 2021 •

edited

Loading

ghost commented Feb 11, 2021 •

edited by ghost

Loading

`Run Window tests`

`Compress`

`Compress`

`Test Sync`

codecov-io commented Feb 11, 2021 •

edited

Loading