Skip to content

[fbreceiver] partial fix for global paths#46779

Merged
leehinman merged 3 commits intoelastic:mainfrom
leehinman:44903_just_filebeat_poc
Sep 29, 2025
Merged

[fbreceiver] partial fix for global paths#46779
leehinman merged 3 commits intoelastic:mainfrom
leehinman:44903_just_filebeat_poc

Conversation

@leehinman
Copy link
Copy Markdown
Contributor

@leehinman leehinman commented Sep 25, 2025 •
edited
Loading

Proposed commit message

** For filebeatreceiver only **

Add a unique per filebeat receiver path variable. All other beats will use the global paths.Paths provided by elastic-agent-libs/paths

This is necessary for each filebeat receiver to have it's own registry in a separate directory.

This is not a complete fix for #44903 . The following paths still need to migrated to use the unique per beat path variable.

  • auditbeat/datastore/datastore.go
  • filebeat/autodiscover/builder/hints/logs.go
  • filebeat/cmd/generate.go
  • libbeat/processors/cache/cache.go
  • libbeat/processors/cache/file_store.go
  • libbeat/processors/script/javascript/javascript.go
  • libbeat/publisher/queue/diskqueue/config.go
  • metricbeat/beater/metricbeat.go
  • winlogbeat/beater/winlogbeat.go
  • x-pack/libbeat/persistentcache/persistentcache.go
  • x-pack/metricbeat/module/openai/usage/usage.go

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

  • [ ]

How to test this PR locally

Unit Test

cd x-pack/filebeat/fbreceiver
go test . -run TestMultipleReceivers -count 1

By hand

  1. Build elastic-agent with this PR for beats dependency
  2. configure multiple filebeat receivers with different path.home
  3. restart multiple times
  4. verify that each filebeat receiver has it's own registry

Related issues

Use cases

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 25, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 25, 2025

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 25, 2025

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @leehinman? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

@leehinman leehinman force-pushed the 44903_just_filebeat_poc branch 3 times, most recently from c273bd9 to 84f3e79 Compare September 26, 2025 14:17
@leehinman leehinman force-pushed the 44903_just_filebeat_poc branch from 84f3e79 to 559b70c Compare September 26, 2025 14:19
@leehinman leehinman marked this pull request as ready for review September 26, 2025 16:52
@leehinman leehinman requested review from a team as code owners September 26, 2025 16:52
@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Sep 26, 2025
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 26, 2025
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 26, 2025

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@pierrehilbert pierrehilbert added Team:obs-ds-hosted-services Label for the Observability Hosted Services team needs_team Indicates that the issue/PR needs a Team:* label labels Sep 26, 2025
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 26, 2025

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 26, 2025
@leehinman leehinman added backport-8.x Automated backport to the 8.x branch with mergify backport-9.1 Automated backport to the 9.1 branch labels Sep 26, 2025
@cmacknz cmacknz mentioned this pull request Sep 26, 2025
Closed
1 task
cmacknz
cmacknz reviewed Sep 26, 2025
View reviewed changes
cmacknz
cmacknz approved these changes Sep 26, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@cmacknz cmacknz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like several of the existing tests had to be modified to account for these changes, so there is test coverage through that.

Do we have any larger test we can rely on to prove we fixed this, so we aren't reliant on manual testing of beats receivers to ensure it doesn't regress.

One wrong use of the global paths import from the wrong source will bring back the problem but there's a lot of surface area to cover. Maybe we can use the linter to forbid the import that would cause this to come back?

cmacknz
cmacknz reviewed Sep 26, 2025
View reviewed changes
@leehinman
Copy link
Copy Markdown
Contributor Author

leehinman commented Sep 26, 2025

Do we have any larger test we can rely on to prove we fixed this, so we aren't reliant on manual testing of beats receivers to ensure it doesn't regress.

The TestMultipleReceivers does make sure we get unique paths, and that they have unique meta.json files.

Could be improved by getting diagnostics from each receiver and making sure the registry in each diagnostic is unique. We could do that in a follow on PR.

One wrong use of the global paths import from the wrong source will bring back the problem but there's a lot of surface area to cover. Maybe we can use the linter to forbid the import that would cause this to come back?

We can deprecate the functions in elastic-agent-libs/paths that use the global variable:

https://github.com/elastic/elastic-agent-libs/blob/b7f794bb30751a6ad6687578482fa0a45171bef4/paths/paths.go#L153

That would let the linter catch any new instances. But we still have to do the work to abolish the global variable.

belimawr
belimawr reviewed Sep 26, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@belimawr belimawr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but having a test to ensure the stores are using different paths would put my mind at easy.

@leehinman
Copy link
Copy Markdown
Contributor Author

leehinman commented Sep 26, 2025

@cmacknz @belimawr I made the "TestMultipleReceivers" more complex, take a look.

faec
faec approved these changes Sep 29, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@faec faec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good as far as this PR, one small related followup for the non-filebeat case

@leehinman leehinman merged commit e55e0be into elastic:main Sep 29, 2025
210 checks passed
mergify bot pushed a commit that referenced this pull request Sep 29, 2025
@leehinman @mergify
[fbreceiver] partial fix for global paths (#46779)
c44d818 
partial fix of global paths for filebeat receiver

(cherry picked from commit e55e0be)

# Conflicts:
#	filebeat/beater/diagnostics.go
#	filebeat/beater/filebeat.go
#	libbeat/cfgfile/reload_integration_test.go
#	libbeat/cmd/instance/beat.go
@mergify mergify bot mentioned this pull request Sep 29, 2025
Closed
6 tasks
@leehinman leehinman mentioned this pull request Oct 1, 2025
Merged
@swiatekm swiatekm mentioned this pull request Dec 1, 2025
Closed
@cmacknz cmacknz added the backport-8.19 Automated backport to the 8.19 branch label Dec 1, 2025
mergify bot pushed a commit that referenced this pull request Dec 1, 2025
@leehinman @mergify
[fbreceiver] partial fix for global paths (#46779)
06fcd33 
partial fix of global paths for filebeat receiver

(cherry picked from commit e55e0be)

# Conflicts:
#	filebeat/beater/crawler.go
#	libbeat/cfgfile/reload_test.go
#	libbeat/cmd/instance/beat.go
#	libbeat/otelbeat/oteltest/oteltest.go
#	x-pack/filebeat/fbreceiver/receiver_test.go
@mergify mergify bot mentioned this pull request Dec 1, 2025
Merged
6 tasks
mauri870 added a commit that referenced this pull request Dec 4, 2025
@mergify @leehinman @mauri870
[8.19](backport #46779) [fbreceiver] partial fix for global paths (#4…
f55d831 
…7829)

* [fbreceiver] partial fix for global paths (#46779)

partial fix of global paths for filebeat receiver

(cherry picked from commit e55e0be)

# Conflicts:
#	filebeat/beater/crawler.go
#	libbeat/cfgfile/reload_test.go
#	libbeat/cmd/instance/beat.go
#	libbeat/otelbeat/oteltest/oteltest.go
#	x-pack/filebeat/fbreceiver/receiver_test.go

* fix conflicts

---------

Co-authored-by: Lee E Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cmacknz cmacknz cmacknz approved these changes

@belimawr belimawr belimawr approved these changes

@Kavindu-Dodan Kavindu-Dodan Kavindu-Dodan approved these changes

@faec faec faec approved these changes

@VihasMakwana VihasMakwana Awaiting requested review from VihasMakwana VihasMakwana is a code owner automatically assigned from elastic/elastic-agent-data-plane

Assignees

@leehinman leehinman

Labels

backport-8.x Automated backport to the 8.x branch with mergify backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Team:obs-ds-hosted-services Label for the Observability Hosted Services team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@leehinman @elasticmachine @belimawr @cmacknz @Kavindu-Dodan @faec @pierrehilbert