otel: add otel-specific fields to ingested docs by mauri870 · Pull Request #45242 · elastic/beats

mauri870 · 2025-07-08T17:24:23Z

Proposed commit message

This PR adds the otelcol.component.id and otelcol.component.kind to ingested beats receivers documents. These fields are exclusive to docs ingested by beats receivers and thus not present for standalone beats. In order to avoid conflicts with logs that already have these fields they are nested inside the beats event agent property.

See https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/rfcs/component-universal-telemetry.md#receivers

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes [OTel] Include receiver-specific fields in ingested Beats documents #45177

elasticmachine · 2025-07-08T17:24:28Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

github-actions · 2025-07-08T17:24:35Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mauri870 · 2025-07-10T19:46:10Z

Leaving this as draft until #45314 is merged, otherwise the tests will keep failing.

mauri870 · 2025-09-01T12:47:16Z

Blocked on #46337.

mauri870 · 2025-09-02T14:29:23Z

/test

mauri870 · 2025-09-03T14:47:03Z

/test

mauri870 · 2025-09-08T13:37:03Z

@elastic/security-service-integrations It looks like I still need a review from your team. Could someone please take a look? Thanks!

efd6

SSI owned file LGTM

github-actions · 2025-09-10T11:04:34Z

@Mergifyio backport 9.0 9.1

mergify · 2025-09-10T11:04:40Z

backport 9.0 9.1

✅ Backports have been created

Details

#46498 [9.0](backport #45242) otel: add otel-specific fields to ingested docs has been created for branch 9.0 but encountered conflicts
#46499 [9.1](backport #45242) otel: add otel-specific fields to ingested docs has been created for branch 9.1 but encountered conflicts

* otel: add otelcol.component.id field to ingested docs * adjust otelcol field name * rename otelcol.component.id in beats event to otel.component.name * add otel.component.type * update metricbeatreceiver unit tests * add integration test coverage * update receiver name in test pipeline * use filestream as receiver name * revert to otelcol.component.id and otelcol.component.kind * fix tests * ignore fields * ignore otel fields in gcppubsub test * move assertion on otel fields to the up * fix linter issues * handle existing otelcol.component.id * prevent otelcol fields from being replaced if they exist * put otelcol fields nested inside agent property * fix E2E tests * use EventuallyWithTf (cherry picked from commit fafbdcb) # Conflicts: # x-pack/filebeat/tests/integration/otel_test.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer_test.go # x-pack/metricbeat/tests/integration/otel_test.go

* otel: add otelcol.component.id field to ingested docs * adjust otelcol field name * rename otelcol.component.id in beats event to otel.component.name * add otel.component.type * update metricbeatreceiver unit tests * add integration test coverage * update receiver name in test pipeline * use filestream as receiver name * revert to otelcol.component.id and otelcol.component.kind * fix tests * ignore fields * ignore otel fields in gcppubsub test * move assertion on otel fields to the up * fix linter issues * handle existing otelcol.component.id * prevent otelcol fields from being replaced if they exist * put otelcol fields nested inside agent property * fix E2E tests * use EventuallyWithTf (cherry picked from commit fafbdcb) # Conflicts: # x-pack/filebeat/fbreceiver/receiver_test.go # x-pack/filebeat/input/gcppubsub/otel_test.go # x-pack/filebeat/tests/integration/otel_test.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer_test.go # x-pack/metricbeat/tests/integration/otel_test.go

* otel: add otelcol.component.id field to ingested docs * adjust otelcol field name * rename otelcol.component.id in beats event to otel.component.name * add otel.component.type * update metricbeatreceiver unit tests * add integration test coverage * update receiver name in test pipeline * use filestream as receiver name * revert to otelcol.component.id and otelcol.component.kind * fix tests * ignore fields * ignore otel fields in gcppubsub test * move assertion on otel fields to the up * fix linter issues * handle existing otelcol.component.id * prevent otelcol fields from being replaced if they exist * put otelcol fields nested inside agent property * fix E2E tests * use EventuallyWithTf (cherry picked from commit fafbdcb) # Conflicts: # x-pack/filebeat/input/gcppubsub/otel_test.go # x-pack/filebeat/tests/integration/otel_test.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer_test.go # x-pack/metricbeat/tests/integration/otel_test.go

#46499) * otel: add otel-specific fields to ingested docs (#45242) * otel: add otelcol.component.id field to ingested docs * adjust otelcol field name * rename otelcol.component.id in beats event to otel.component.name * add otel.component.type * update metricbeatreceiver unit tests * add integration test coverage * update receiver name in test pipeline * use filestream as receiver name * revert to otelcol.component.id and otelcol.component.kind * fix tests * ignore fields * ignore otel fields in gcppubsub test * move assertion on otel fields to the up * fix linter issues * handle existing otelcol.component.id * prevent otelcol fields from being replaced if they exist * put otelcol fields nested inside agent property * fix E2E tests * use EventuallyWithTf (cherry picked from commit fafbdcb) # Conflicts: # x-pack/filebeat/input/gcppubsub/otel_test.go # x-pack/filebeat/tests/integration/otel_test.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer_test.go # x-pack/metricbeat/tests/integration/otel_test.go * fix conflicts * remove test not in the 9.1 branch * remove more tests not in base branch * remove duplicated client import --------- Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>

…cs (#46497) * otel: add otel-specific fields to ingested docs (#45242) * otel: add otelcol.component.id field to ingested docs * adjust otelcol field name * rename otelcol.component.id in beats event to otel.component.name * add otel.component.type * update metricbeatreceiver unit tests * add integration test coverage * update receiver name in test pipeline * use filestream as receiver name * revert to otelcol.component.id and otelcol.component.kind * fix tests * ignore fields * ignore otel fields in gcppubsub test * move assertion on otel fields to the up * fix linter issues * handle existing otelcol.component.id * prevent otelcol fields from being replaced if they exist * put otelcol fields nested inside agent property * fix E2E tests * use EventuallyWithTf (cherry picked from commit fafbdcb) # Conflicts: # x-pack/filebeat/tests/integration/otel_test.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer_test.go # x-pack/metricbeat/tests/integration/otel_test.go * fix conflicts * reorder variables in otelconsumer --------- Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>

* otel: add otelcol.component.id field to ingested docs * adjust otelcol field name * rename otelcol.component.id in beats event to otel.component.name * add otel.component.type * update metricbeatreceiver unit tests * add integration test coverage * update receiver name in test pipeline * use filestream as receiver name * revert to otelcol.component.id and otelcol.component.kind * fix tests * ignore fields * ignore otel fields in gcppubsub test * move assertion on otel fields to the up * fix linter issues * handle existing otelcol.component.id * prevent otelcol fields from being replaced if they exist * put otelcol fields nested inside agent property * fix E2E tests * use EventuallyWithTf

mauri870 self-assigned this Jul 8, 2025

mauri870 requested a review from a team as a code owner July 8, 2025 17:24

mauri870 added cleanup Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Jul 8, 2025

mauri870 requested review from faec and khushijain21 July 8, 2025 17:24

mauri870 added backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches backport-8.19 Automated backport to the 8.19 branch labels Jul 8, 2025

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 8, 2025

mauri870 marked this pull request as draft July 8, 2025 20:21

mauri870 force-pushed the otel-component-id branch from acde212 to c8eab70 Compare July 8, 2025 20:24

mauri870 marked this pull request as ready for review July 8, 2025 20:30

mauri870 changed the title ~~otel: add otelcol.component.id field to ingested docs~~ otel: add otel-specific fields to ingested docs Jul 10, 2025

mauri870 mentioned this pull request Jul 10, 2025

otel: update filebeat integration tests to run in isolation #45314

Merged

6 tasks

mauri870 marked this pull request as draft July 10, 2025 19:46

khushijain21 reviewed Jul 11, 2025

View reviewed changes

Comment thread libbeat/beat/info.go Outdated

mauri870 added 7 commits July 15, 2025 13:19

otel: add otelcol.component.id field to ingested docs

fa7b90c

adjust otelcol field name

6ba3880

rename otelcol.component.id in beats event to otel.component.name

ce990d1

add otel.component.type

7b67ee9

update metricbeatreceiver unit tests

9a6f3a1

add integration test coverage

67939f5

update receiver name in test pipeline

f310226

ishleenk17 reviewed Sep 1, 2025

View reviewed changes

Comment thread x-pack/filebeat/input/gcppubsub/otel_test.go

mauri870 requested a review from ishleenk17 September 1, 2025 15:28

Merge branch 'main' into otel-component-id

0b3ba27

Merge branch 'main' into otel-component-id

a8743ac

mauri870 added 6 commits September 3, 2025 12:58

Merge branch 'main' into otel-component-id

8b9d402

Merge branch 'main' into otel-component-id

1f59834

Merge branch 'main' into otel-component-id

2353d18

Merge branch 'main' into otel-component-id

3fa9a89

Merge branch 'main' into otel-component-id

ea52083

Merge branch 'main' into otel-component-id

d57ebe5

mauri870 added the backport-8.19 Automated backport to the 8.19 branch label Sep 9, 2025

efd6 approved these changes Sep 10, 2025

View reviewed changes

mauri870 merged commit fafbdcb into elastic:main Sep 10, 2025
208 checks passed

mergify bot mentioned this pull request Sep 10, 2025

[8.19](backport #45242) otel: add otel-specific fields to ingested docs #46497

Merged

6 tasks

mergify bot mentioned this pull request Sep 10, 2025

[9.0](backport #45242) otel: add otel-specific fields to ingested docs #46498

Closed

6 tasks

mergify bot mentioned this pull request Sep 10, 2025

[9.1](backport #45242) otel: add otel-specific fields to ingested docs #46499

Merged

6 tasks

mauri870 mentioned this pull request Sep 11, 2025

[main][Automation] Update elastic/beats to afc53c0479ac elastic/elastic-agent#9874

Merged

cmacknz mentioned this pull request Nov 12, 2025

[beats receivers] Remove collector component identifying fields #47600

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

otel: add otel-specific fields to ingested docs#45242

otel: add otel-specific fields to ingested docs#45242
mauri870 merged 44 commits intoelastic:mainfrom
mauri870:otel-component-id

mauri870 commented Jul 8, 2025 •

edited

Loading

Uh oh!

elasticmachine commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

mauri870 commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!

mauri870 commented Sep 1, 2025

Uh oh!

mauri870 commented Sep 2, 2025

Uh oh!

mauri870 commented Sep 3, 2025

Uh oh!

mauri870 commented Sep 8, 2025

Uh oh!

efd6 left a comment

Uh oh!

Uh oh!

github-actions bot commented Sep 10, 2025

Uh oh!

mergify bot commented Sep 10, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

mauri870 commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Related issues

Uh oh!

elasticmachine commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

🤖 GitHub comments

Uh oh!

mauri870 commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!

mauri870 commented Sep 1, 2025

Uh oh!

mauri870 commented Sep 2, 2025

Uh oh!

mauri870 commented Sep 3, 2025

Uh oh!

mauri870 commented Sep 8, 2025

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Sep 10, 2025

Uh oh!

mergify bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mauri870 commented Jul 8, 2025 •

edited

Loading

mergify bot commented Sep 10, 2025 •

edited

Loading