Skip to content

[9.3](backport #48834) fix(diskqueue): use per-beat paths instead of global paths#49276

Merged
orestisfl merged 1 commit into9.3from
mergify/bp/9.3/pr-48834
Mar 5, 2026
Merged

[9.3](backport #48834) fix(diskqueue): use per-beat paths instead of global paths#49276
orestisfl merged 1 commit into9.3from
mergify/bp/9.3/pr-48834

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Mar 4, 2026

Proposed commit message

fix(diskqueue): use per-beat paths instead of global paths

Add a Paths field to diskqueue Settings so each beat receiver gets its own data directory for the disk queue. Thread the per-beat *paths.Path through the pipeline and output factory APIs:

  • outputs.Factory, outputs.Load, outputs.Success, outputs.SuccessNet now accept *paths.Path
  • pipeline.Settings carries Paths and forwards it to queueFactoryForUserConfig
  • All 8 output factory implementations (console, discard, elasticsearch, file, kafka, logstash, redis, otelconsumer) updated to accept and forward paths
  • dockerlogbeat updated to pass paths.New() explicitly. This is making explicit the previously implied behavior: the global paths were never initialized or used in dockerlogbeat.
  • pipeline/stress test harness updated to use LoadWithSettings with paths

When Settings.Path is explicitly configured, it still takes precedence (backward compatible). When empty, the diskqueue directory is resolved via the per-beat Paths.Resolve() instead of the global paths.Resolve().

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
  • I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None. In the standalone beat flow, b.Paths and paths.Paths point to the same object, so behavior is identical. The change only corrects behavior for beat receivers, where each receiver gets its own paths instance.

How to test this PR locally

Standalone filebeat

# filebeat-diskqueue-test.yml
path.home: /tmp/diskqueue-test/standalone
path.data: /tmp/diskqueue-test/standalone/data

filebeat.inputs:
  - type: filestream
    id: test-input
    enabled: true
    paths:
      - /tmp/diskqueue-test/input.log
    prospector.scanner.fingerprint.enabled: false
    file_identity.native: ~

queue.disk:
  max_size: 100MB

output.console:
  pretty: true
cd filebeat && mage build
echo '{"message": "hello"}' > /tmp/diskqueue-test/input.log
./filebeat -e -c /path/to/filebeat-diskqueue-test.yml

# Verify: diskqueue created under path.data
ls -la /tmp/diskqueue-test/standalone/data/diskqueue/
# Should contain: state.dat (and .seg files once events flow)

elastic-agent otel

Create /tmp/diskqueue-test/otel-receivers.yml:

receivers:
  filebeatreceiver/a:
    filebeat:
      inputs:
        - type: filestream
          id: stream-a
          enabled: true
          paths:
            - /tmp/diskqueue-test/input.log
          prospector.scanner.fingerprint.enabled: false
          file_identity.native: ~
    path.home: /tmp/diskqueue-test/receiver-a
    queue.disk:
      max_size: 100MB

  filebeatreceiver/b:
    filebeat:
      inputs:
        - type: filestream
          id: stream-b
          enabled: true
          paths:
            - /tmp/diskqueue-test/input.log
          prospector.scanner.fingerprint.enabled: false
          file_identity.native: ~
    path.home: /tmp/diskqueue-test/receiver-b
    queue.disk:
      max_size: 100MB

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filebeatreceiver/a, filebeatreceiver/b]
      exporters: [debug]
  telemetry:
    metrics:
      level: none

Run:

elastic-agent otel --config /tmp/diskqueue-test/otel-receivers.yml

Verify:

find /tmp/diskqueue-test/receiver-a -name state.dat
# Expected: /tmp/diskqueue-test/receiver-a/data/diskqueue/state.dat

find /tmp/diskqueue-test/receiver-b -name state.dat
# Expected: /tmp/diskqueue-test/receiver-b/data/diskqueue/state.dat

Related issues


This is an automatic backport of pull request #48834 done by [Mergify](https://mergify.com).

fix(diskqueue): use per-beat paths instead of global paths

Add a `Paths` field to diskqueue `Settings` so each beat receiver gets its own data directory for the disk queue. Thread the per-beat `*paths.Path` through the pipeline and output factory APIs:

- `outputs.Factory`, `outputs.Load`, `outputs.Success`, `outputs.SuccessNet` now accept `*paths.Path`
- `pipeline.Settings` carries `Paths` and forwards it to `queueFactoryForUserConfig`
- All 8 output factory implementations (console, discard, elasticsearch, file, kafka, logstash, redis, otelconsumer) updated to accept and forward paths
- `dockerlogbeat` updated to pass `paths.New()` explicitly. This is making explicit the previously implied behavior: the global paths were never initialized or used in `dockerlogbeat`.
- `pipeline/stress` test harness updated to use `LoadWithSettings` with paths

When `Settings.Path` is explicitly configured, it still takes precedence (backward compatible). When empty, the diskqueue directory is resolved via the per-beat `Paths.Resolve()` instead of the global `paths.Resolve()`.

## Checklist

- [x] My code follows the style guidelines of this project
- [x] I have commented my code, particularly in hard-to-understand areas
- ~~I have made corresponding changes to the documentation~~
- ~~I have made corresponding change to the default configuration files~~
- [x] I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the [`stresstest.sh`](https://github.com/elastic/beats/blob/main/script/stresstest.sh) script to run them under stress conditions and race detector to verify their stability.
- ~~I have added an entry in `./changelog/fragments` using the [changelog tool](https://github.com/elastic/elastic-agent-changelog-tool/blob/main/docs/usage.md).~~

## Disruptive User Impact

None. In the standalone beat flow, `b.Paths` and `paths.Paths` point to the same object, so behavior is identical. The change only corrects behavior for beat receivers, where each receiver gets its own paths instance.

## How to test this PR locally
### Standalone filebeat

```yaml
# filebeat-diskqueue-test.yml
path.home: /tmp/diskqueue-test/standalone
path.data: /tmp/diskqueue-test/standalone/data

filebeat.inputs:
  - type: filestream
    id: test-input
    enabled: true
    paths:
      - /tmp/diskqueue-test/input.log
    prospector.scanner.fingerprint.enabled: false
    file_identity.native: ~

queue.disk:
  max_size: 100MB

output.console:
  pretty: true
```

```bash
cd filebeat && mage build
echo '{"message": "hello"}' > /tmp/diskqueue-test/input.log
./filebeat -e -c /path/to/filebeat-diskqueue-test.yml

# Verify: diskqueue created under path.data
ls -la /tmp/diskqueue-test/standalone/data/diskqueue/
# Should contain: state.dat (and .seg files once events flow)
```

### `elastic-agent otel`
Create `/tmp/diskqueue-test/otel-receivers.yml`:

```yaml
receivers:
  filebeatreceiver/a:
    filebeat:
      inputs:
        - type: filestream
          id: stream-a
          enabled: true
          paths:
            - /tmp/diskqueue-test/input.log
          prospector.scanner.fingerprint.enabled: false
          file_identity.native: ~
    path.home: /tmp/diskqueue-test/receiver-a
    queue.disk:
      max_size: 100MB

  filebeatreceiver/b:
    filebeat:
      inputs:
        - type: filestream
          id: stream-b
          enabled: true
          paths:
            - /tmp/diskqueue-test/input.log
          prospector.scanner.fingerprint.enabled: false
          file_identity.native: ~
    path.home: /tmp/diskqueue-test/receiver-b
    queue.disk:
      max_size: 100MB

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filebeatreceiver/a, filebeatreceiver/b]
      exporters: [debug]
  telemetry:
    metrics:
      level: none
```

Run:

```bash
elastic-agent otel --config /tmp/diskqueue-test/otel-receivers.yml
```

Verify:

```bash
find /tmp/diskqueue-test/receiver-a -name state.dat
# Expected: /tmp/diskqueue-test/receiver-a/data/diskqueue/state.dat

find /tmp/diskqueue-test/receiver-b -name state.dat
# Expected: /tmp/diskqueue-test/receiver-b/data/diskqueue/state.dat
```

## Related issues

- Closes #46989

(cherry picked from commit 979f9b4)
@mergify mergify bot added the backport label Mar 4, 2026
@mergify mergify bot requested a review from a team as a code owner March 4, 2026 15:11
@mergify mergify bot requested review from AndersonQ and VihasMakwana and removed request for a team March 4, 2026 15:11
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 4, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 4, 2026

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@github-actions github-actions bot added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team skip-changelog labels Mar 4, 2026
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 4, 2026
@orestisfl orestisfl merged commit 77ff8b8 into 9.3 Mar 5, 2026
207 of 210 checks passed
@orestisfl orestisfl deleted the mergify/bp/9.3/pr-48834 branch March 5, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants