[9.3](backport #48834) fix(diskqueue): use per-beat paths instead of global paths#49276
Merged
[9.3](backport #48834) fix(diskqueue): use per-beat paths instead of global paths#49276
Conversation
fix(diskqueue): use per-beat paths instead of global paths Add a `Paths` field to diskqueue `Settings` so each beat receiver gets its own data directory for the disk queue. Thread the per-beat `*paths.Path` through the pipeline and output factory APIs: - `outputs.Factory`, `outputs.Load`, `outputs.Success`, `outputs.SuccessNet` now accept `*paths.Path` - `pipeline.Settings` carries `Paths` and forwards it to `queueFactoryForUserConfig` - All 8 output factory implementations (console, discard, elasticsearch, file, kafka, logstash, redis, otelconsumer) updated to accept and forward paths - `dockerlogbeat` updated to pass `paths.New()` explicitly. This is making explicit the previously implied behavior: the global paths were never initialized or used in `dockerlogbeat`. - `pipeline/stress` test harness updated to use `LoadWithSettings` with paths When `Settings.Path` is explicitly configured, it still takes precedence (backward compatible). When empty, the diskqueue directory is resolved via the per-beat `Paths.Resolve()` instead of the global `paths.Resolve()`. ## Checklist - [x] My code follows the style guidelines of this project - [x] I have commented my code, particularly in hard-to-understand areas - ~~I have made corresponding changes to the documentation~~ - ~~I have made corresponding change to the default configuration files~~ - [x] I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the [`stresstest.sh`](https://github.com/elastic/beats/blob/main/script/stresstest.sh) script to run them under stress conditions and race detector to verify their stability. - ~~I have added an entry in `./changelog/fragments` using the [changelog tool](https://github.com/elastic/elastic-agent-changelog-tool/blob/main/docs/usage.md).~~ ## Disruptive User Impact None. In the standalone beat flow, `b.Paths` and `paths.Paths` point to the same object, so behavior is identical. The change only corrects behavior for beat receivers, where each receiver gets its own paths instance. ## How to test this PR locally ### Standalone filebeat ```yaml # filebeat-diskqueue-test.yml path.home: /tmp/diskqueue-test/standalone path.data: /tmp/diskqueue-test/standalone/data filebeat.inputs: - type: filestream id: test-input enabled: true paths: - /tmp/diskqueue-test/input.log prospector.scanner.fingerprint.enabled: false file_identity.native: ~ queue.disk: max_size: 100MB output.console: pretty: true ``` ```bash cd filebeat && mage build echo '{"message": "hello"}' > /tmp/diskqueue-test/input.log ./filebeat -e -c /path/to/filebeat-diskqueue-test.yml # Verify: diskqueue created under path.data ls -la /tmp/diskqueue-test/standalone/data/diskqueue/ # Should contain: state.dat (and .seg files once events flow) ``` ### `elastic-agent otel` Create `/tmp/diskqueue-test/otel-receivers.yml`: ```yaml receivers: filebeatreceiver/a: filebeat: inputs: - type: filestream id: stream-a enabled: true paths: - /tmp/diskqueue-test/input.log prospector.scanner.fingerprint.enabled: false file_identity.native: ~ path.home: /tmp/diskqueue-test/receiver-a queue.disk: max_size: 100MB filebeatreceiver/b: filebeat: inputs: - type: filestream id: stream-b enabled: true paths: - /tmp/diskqueue-test/input.log prospector.scanner.fingerprint.enabled: false file_identity.native: ~ path.home: /tmp/diskqueue-test/receiver-b queue.disk: max_size: 100MB exporters: debug: verbosity: detailed service: pipelines: logs: receivers: [filebeatreceiver/a, filebeatreceiver/b] exporters: [debug] telemetry: metrics: level: none ``` Run: ```bash elastic-agent otel --config /tmp/diskqueue-test/otel-receivers.yml ``` Verify: ```bash find /tmp/diskqueue-test/receiver-a -name state.dat # Expected: /tmp/diskqueue-test/receiver-a/data/diskqueue/state.dat find /tmp/diskqueue-test/receiver-b -name state.dat # Expected: /tmp/diskqueue-test/receiver-b/data/diskqueue/state.dat ``` ## Related issues - Closes #46989 (cherry picked from commit 979f9b4)
Contributor
🤖 GitHub commentsJust comment with:
|
Contributor
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
orestisfl
approved these changes
Mar 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
fix(diskqueue): use per-beat paths instead of global paths
Add a
Pathsfield to diskqueueSettingsso each beat receiver gets its own data directory for the disk queue. Thread the per-beat*paths.Paththrough the pipeline and output factory APIs:outputs.Factory,outputs.Load,outputs.Success,outputs.SuccessNetnow accept*paths.Pathpipeline.SettingscarriesPathsand forwards it toqueueFactoryForUserConfigdockerlogbeatupdated to passpaths.New()explicitly. This is making explicit the previously implied behavior: the global paths were never initialized or used indockerlogbeat.pipeline/stresstest harness updated to useLoadWithSettingswith pathsWhen
Settings.Pathis explicitly configured, it still takes precedence (backward compatible). When empty, the diskqueue directory is resolved via the per-beatPaths.Resolve()instead of the globalpaths.Resolve().Checklist
I have made corresponding changes to the documentationI have made corresponding change to the default configuration filesstresstest.shscript to run them under stress conditions and race detector to verify their stability.I have added an entry in./changelog/fragmentsusing the changelog tool.Disruptive User Impact
None. In the standalone beat flow,
b.Pathsandpaths.Pathspoint to the same object, so behavior is identical. The change only corrects behavior for beat receivers, where each receiver gets its own paths instance.How to test this PR locally
Standalone filebeat
elastic-agent otelCreate
/tmp/diskqueue-test/otel-receivers.yml:Run:
Verify:
Related issues
This is an automatic backport of pull request #48834 done by [Mergify](https://mergify.com).