[8.19](backport #48834) fix(diskqueue): use per-beat paths instead of global paths#49274
[8.19](backport #48834) fix(diskqueue): use per-beat paths instead of global paths#49274
Conversation
fix(diskqueue): use per-beat paths instead of global paths Add a `Paths` field to diskqueue `Settings` so each beat receiver gets its own data directory for the disk queue. Thread the per-beat `*paths.Path` through the pipeline and output factory APIs: - `outputs.Factory`, `outputs.Load`, `outputs.Success`, `outputs.SuccessNet` now accept `*paths.Path` - `pipeline.Settings` carries `Paths` and forwards it to `queueFactoryForUserConfig` - All 8 output factory implementations (console, discard, elasticsearch, file, kafka, logstash, redis, otelconsumer) updated to accept and forward paths - `dockerlogbeat` updated to pass `paths.New()` explicitly. This is making explicit the previously implied behavior: the global paths were never initialized or used in `dockerlogbeat`. - `pipeline/stress` test harness updated to use `LoadWithSettings` with paths When `Settings.Path` is explicitly configured, it still takes precedence (backward compatible). When empty, the diskqueue directory is resolved via the per-beat `Paths.Resolve()` instead of the global `paths.Resolve()`. ## Checklist - [x] My code follows the style guidelines of this project - [x] I have commented my code, particularly in hard-to-understand areas - ~~I have made corresponding changes to the documentation~~ - ~~I have made corresponding change to the default configuration files~~ - [x] I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the [`stresstest.sh`](https://github.com/elastic/beats/blob/main/script/stresstest.sh) script to run them under stress conditions and race detector to verify their stability. - ~~I have added an entry in `./changelog/fragments` using the [changelog tool](https://github.com/elastic/elastic-agent-changelog-tool/blob/main/docs/usage.md).~~ ## Disruptive User Impact None. In the standalone beat flow, `b.Paths` and `paths.Paths` point to the same object, so behavior is identical. The change only corrects behavior for beat receivers, where each receiver gets its own paths instance. ## How to test this PR locally ### Standalone filebeat ```yaml # filebeat-diskqueue-test.yml path.home: /tmp/diskqueue-test/standalone path.data: /tmp/diskqueue-test/standalone/data filebeat.inputs: - type: filestream id: test-input enabled: true paths: - /tmp/diskqueue-test/input.log prospector.scanner.fingerprint.enabled: false file_identity.native: ~ queue.disk: max_size: 100MB output.console: pretty: true ``` ```bash cd filebeat && mage build echo '{"message": "hello"}' > /tmp/diskqueue-test/input.log ./filebeat -e -c /path/to/filebeat-diskqueue-test.yml # Verify: diskqueue created under path.data ls -la /tmp/diskqueue-test/standalone/data/diskqueue/ # Should contain: state.dat (and .seg files once events flow) ``` ### `elastic-agent otel` Create `/tmp/diskqueue-test/otel-receivers.yml`: ```yaml receivers: filebeatreceiver/a: filebeat: inputs: - type: filestream id: stream-a enabled: true paths: - /tmp/diskqueue-test/input.log prospector.scanner.fingerprint.enabled: false file_identity.native: ~ path.home: /tmp/diskqueue-test/receiver-a queue.disk: max_size: 100MB filebeatreceiver/b: filebeat: inputs: - type: filestream id: stream-b enabled: true paths: - /tmp/diskqueue-test/input.log prospector.scanner.fingerprint.enabled: false file_identity.native: ~ path.home: /tmp/diskqueue-test/receiver-b queue.disk: max_size: 100MB exporters: debug: verbosity: detailed service: pipelines: logs: receivers: [filebeatreceiver/a, filebeatreceiver/b] exporters: [debug] telemetry: metrics: level: none ``` Run: ```bash elastic-agent otel --config /tmp/diskqueue-test/otel-receivers.yml ``` Verify: ```bash find /tmp/diskqueue-test/receiver-a -name state.dat # Expected: /tmp/diskqueue-test/receiver-a/data/diskqueue/state.dat find /tmp/diskqueue-test/receiver-b -name state.dat # Expected: /tmp/diskqueue-test/receiver-b/data/diskqueue/state.dat ``` ## Related issues - Closes #46989 (cherry picked from commit 979f9b4) # Conflicts: # libbeat/outputs/logstash/logstash.go # x-pack/otel/exporter/logstashexporter/exporter.go
|
Cherry-pick of 979f9b4 has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
🤖 GitHub commentsJust comment with:
|
|
This pull request has not been merged yet. Could you please review and merge it @orestisfl? 🙏 |
Resolve cherry-pick conflicts from #48834 backport to 8.19: - logstash.go: keep 8.19 code structure (no MakeLogstashClients refactoring), add paths import and pass beatPaths to SuccessNet - exporter.go: remove file that doesn't exist on 8.19 (the logstashexporter package is a main-branch feature)
|
The failure is in Summary of failing checks from the logs:
Recommended minimal remediation:
Tests run by this investigation:
What is this? | From workflow: PR Actions Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
|
The failure is in golangci-lint ( What failed
Concrete failing locations from logs
Minimal remediation plan
Tests run
What is this? | From workflow: PR Actions Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
Proposed commit message
fix(diskqueue): use per-beat paths instead of global paths
Add a
Pathsfield to diskqueueSettingsso each beat receiver gets its own data directory for the disk queue. Thread the per-beat*paths.Paththrough the pipeline and output factory APIs:outputs.Factory,outputs.Load,outputs.Success,outputs.SuccessNetnow accept*paths.Pathpipeline.SettingscarriesPathsand forwards it toqueueFactoryForUserConfigdockerlogbeatupdated to passpaths.New()explicitly. This is making explicit the previously implied behavior: the global paths were never initialized or used indockerlogbeat.pipeline/stresstest harness updated to useLoadWithSettingswith pathsWhen
Settings.Pathis explicitly configured, it still takes precedence (backward compatible). When empty, the diskqueue directory is resolved via the per-beatPaths.Resolve()instead of the globalpaths.Resolve().Checklist
I have made corresponding changes to the documentationI have made corresponding change to the default configuration filesstresstest.shscript to run them under stress conditions and race detector to verify their stability.I have added an entry in./changelog/fragmentsusing the changelog tool.Disruptive User Impact
None. In the standalone beat flow,
b.Pathsandpaths.Pathspoint to the same object, so behavior is identical. The change only corrects behavior for beat receivers, where each receiver gets its own paths instance.How to test this PR locally
Standalone filebeat
elastic-agent otelCreate
/tmp/diskqueue-test/otel-receivers.yml:Run:
Verify:
Related issues
This is an automatic backport of pull request #48834 done by [Mergify](https://mergify.com).