Skip to content

Remap OTel hostmetrics to Elastic metrics#277

Merged
lahsivjar merged 5 commits intoelastic:mainfrom
lahsivjar:hostmetrics-poc
Jun 5, 2024
Merged

Remap OTel hostmetrics to Elastic metrics#277
lahsivjar merged 5 commits intoelastic:mainfrom
lahsivjar:hostmetrics-poc

Conversation

@lahsivjar
Copy link
Copy Markdown
Contributor

@lahsivjar lahsivjar commented May 9, 2024 •
edited
Loading

The PR uses the library opentelemetry-lib to remap metrics produced by hostmetricsreceiver to Elastic metrics that could power the curated UIs.

Example OTel collector configuration required to get the remapping working 
receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      load:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
          system.cpu.logical.count:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      network:
      process:
        metrics:
          process.threads:
            enabled: true
          process.open_file_descriptors:
            enabled: true
          process.memory.utilization:
            enabled: true
          process.disk.operations:
            enabled: true
      processes:

exporters:
  otlphttp:
    endpoint: http://192.168.0.44:8200
    tls:
      insecure: true
  debug:
    verbosity: detailed

processors:
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection/system]
      exporters: [otlphttp, debug]

Related issues

@elastic-apm-tech elastic-apm-tech added the safe-to-test Changes are safe to run in the CI label May 9, 2024
@lahsivjar lahsivjar force-pushed the hostmetrics-poc branch 2 times, most recently from 0f968a9 to 2ae4207 Compare May 20, 2024 21:27
@lahsivjar lahsivjar changed the title Remap cpu, memory, process, and load scrapers for OTel hostmetrics Remap OTel hostmetrics to Elastic metrics May 20, 2024
@lahsivjar lahsivjar force-pushed the hostmetrics-poc branch 2 times, most recently from 7466b40 to 3e64024 Compare May 20, 2024 21:36
@lahsivjar lahsivjar mentioned this pull request May 20, 2024
Merged
1 task
@lahsivjar lahsivjar marked this pull request as ready for review May 29, 2024 13:54
@lahsivjar lahsivjar requested a review from a team as a code owner May 29, 2024 13:54
lahsivjar
lahsivjar commented May 29, 2024
View reviewed changes
model/modelprocessor/datastream.go
Comment on lines +144 to +155

// set internal to false for metrics translated using OTel remappers.
if label, ok := event.Labels["event.provider"]; ok && label != nil {
internal = !(label.Value == "hostmetrics")
}

if internal {
for _, s := range event.Metricset.Samples {
if !IsInternalMetricName(s.Name) {
internal = false
break
}
Copy link
Copy Markdown
Contributor Author

@lahsivjar lahsivjar May 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[For reviewers] This code is required to make sure that the remapped OTel metrics end up in the app datastream instead of the internal datastream.

kruskall
kruskall previously approved these changes May 29, 2024
View reviewed changes
endorama
endorama reviewed May 29, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@endorama endorama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a blocking comment and some minor non-blocking discussion points.

@lahsivjar lahsivjar requested a review from endorama May 29, 2024 20:13
axw
axw reviewed May 30, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@axw axw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM, just wondering about the event.provider check. I think we should come to an agreement on that before merging.

@lahsivjar lahsivjar requested a review from axw June 4, 2024 14:40
endorama
endorama previously approved these changes Jun 4, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@endorama endorama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

axw
axw previously approved these changes Jun 5, 2024
View reviewed changes
@lahsivjar @axw
Update input/otlp/metrics.go
f686b33 
Co-authored-by: Andrew Wilkins <axwalk@gmail.com>
@lahsivjar lahsivjar dismissed stale reviews from axw and endorama via f686b33 June 5, 2024 08:03
@lahsivjar lahsivjar requested review from axw and endorama June 5, 2024 08:03
@lahsivjar lahsivjar enabled auto-merge (squash) June 5, 2024 08:03
@lahsivjar lahsivjar merged commit 254ea92 into elastic:main Jun 5, 2024
@lahsivjar lahsivjar deleted the hostmetrics-poc branch June 5, 2024 08:43
@lahsivjar lahsivjar mentioned this pull request Jun 18, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@axw axw axw approved these changes

@endorama endorama Awaiting requested review from endorama

+1 more reviewer

@kruskall kruskall kruskall left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

safe-to-test Changes are safe to run in the CI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@lahsivjar @endorama @axw @kruskall @elastic-apm-tech