Skip to content

[8.19](backport #48834) fix(diskqueue): use per-beat paths instead of global paths#49274

Merged
orestisfl merged 5 commits into8.19from
mergify/bp/8.19/pr-48834
Mar 11, 2026
Merged

[8.19](backport #48834) fix(diskqueue): use per-beat paths instead of global paths#49274
orestisfl merged 5 commits into8.19from
mergify/bp/8.19/pr-48834

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Mar 4, 2026

Proposed commit message

fix(diskqueue): use per-beat paths instead of global paths

Add a Paths field to diskqueue Settings so each beat receiver gets its own data directory for the disk queue. Thread the per-beat *paths.Path through the pipeline and output factory APIs:

  • outputs.Factory, outputs.Load, outputs.Success, outputs.SuccessNet now accept *paths.Path
  • pipeline.Settings carries Paths and forwards it to queueFactoryForUserConfig
  • All 8 output factory implementations (console, discard, elasticsearch, file, kafka, logstash, redis, otelconsumer) updated to accept and forward paths
  • dockerlogbeat updated to pass paths.New() explicitly. This is making explicit the previously implied behavior: the global paths were never initialized or used in dockerlogbeat.
  • pipeline/stress test harness updated to use LoadWithSettings with paths

When Settings.Path is explicitly configured, it still takes precedence (backward compatible). When empty, the diskqueue directory is resolved via the per-beat Paths.Resolve() instead of the global paths.Resolve().

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
  • I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None. In the standalone beat flow, b.Paths and paths.Paths point to the same object, so behavior is identical. The change only corrects behavior for beat receivers, where each receiver gets its own paths instance.

How to test this PR locally

Standalone filebeat

# filebeat-diskqueue-test.yml
path.home: /tmp/diskqueue-test/standalone
path.data: /tmp/diskqueue-test/standalone/data

filebeat.inputs:
  - type: filestream
    id: test-input
    enabled: true
    paths:
      - /tmp/diskqueue-test/input.log
    prospector.scanner.fingerprint.enabled: false
    file_identity.native: ~

queue.disk:
  max_size: 100MB

output.console:
  pretty: true
cd filebeat && mage build
echo '{"message": "hello"}' > /tmp/diskqueue-test/input.log
./filebeat -e -c /path/to/filebeat-diskqueue-test.yml

# Verify: diskqueue created under path.data
ls -la /tmp/diskqueue-test/standalone/data/diskqueue/
# Should contain: state.dat (and .seg files once events flow)

elastic-agent otel

Create /tmp/diskqueue-test/otel-receivers.yml:

receivers:
  filebeatreceiver/a:
    filebeat:
      inputs:
        - type: filestream
          id: stream-a
          enabled: true
          paths:
            - /tmp/diskqueue-test/input.log
          prospector.scanner.fingerprint.enabled: false
          file_identity.native: ~
    path.home: /tmp/diskqueue-test/receiver-a
    queue.disk:
      max_size: 100MB

  filebeatreceiver/b:
    filebeat:
      inputs:
        - type: filestream
          id: stream-b
          enabled: true
          paths:
            - /tmp/diskqueue-test/input.log
          prospector.scanner.fingerprint.enabled: false
          file_identity.native: ~
    path.home: /tmp/diskqueue-test/receiver-b
    queue.disk:
      max_size: 100MB

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filebeatreceiver/a, filebeatreceiver/b]
      exporters: [debug]
  telemetry:
    metrics:
      level: none

Run:

elastic-agent otel --config /tmp/diskqueue-test/otel-receivers.yml

Verify:

find /tmp/diskqueue-test/receiver-a -name state.dat
# Expected: /tmp/diskqueue-test/receiver-a/data/diskqueue/state.dat

find /tmp/diskqueue-test/receiver-b -name state.dat
# Expected: /tmp/diskqueue-test/receiver-b/data/diskqueue/state.dat

Related issues


This is an automatic backport of pull request #48834 done by [Mergify](https://mergify.com).

fix(diskqueue): use per-beat paths instead of global paths

Add a `Paths` field to diskqueue `Settings` so each beat receiver gets its own data directory for the disk queue. Thread the per-beat `*paths.Path` through the pipeline and output factory APIs:

- `outputs.Factory`, `outputs.Load`, `outputs.Success`, `outputs.SuccessNet` now accept `*paths.Path`
- `pipeline.Settings` carries `Paths` and forwards it to `queueFactoryForUserConfig`
- All 8 output factory implementations (console, discard, elasticsearch, file, kafka, logstash, redis, otelconsumer) updated to accept and forward paths
- `dockerlogbeat` updated to pass `paths.New()` explicitly. This is making explicit the previously implied behavior: the global paths were never initialized or used in `dockerlogbeat`.
- `pipeline/stress` test harness updated to use `LoadWithSettings` with paths

When `Settings.Path` is explicitly configured, it still takes precedence (backward compatible). When empty, the diskqueue directory is resolved via the per-beat `Paths.Resolve()` instead of the global `paths.Resolve()`.

## Checklist

- [x] My code follows the style guidelines of this project
- [x] I have commented my code, particularly in hard-to-understand areas
- ~~I have made corresponding changes to the documentation~~
- ~~I have made corresponding change to the default configuration files~~
- [x] I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the [`stresstest.sh`](https://github.com/elastic/beats/blob/main/script/stresstest.sh) script to run them under stress conditions and race detector to verify their stability.
- ~~I have added an entry in `./changelog/fragments` using the [changelog tool](https://github.com/elastic/elastic-agent-changelog-tool/blob/main/docs/usage.md).~~

## Disruptive User Impact

None. In the standalone beat flow, `b.Paths` and `paths.Paths` point to the same object, so behavior is identical. The change only corrects behavior for beat receivers, where each receiver gets its own paths instance.

## How to test this PR locally
### Standalone filebeat

```yaml
# filebeat-diskqueue-test.yml
path.home: /tmp/diskqueue-test/standalone
path.data: /tmp/diskqueue-test/standalone/data

filebeat.inputs:
  - type: filestream
    id: test-input
    enabled: true
    paths:
      - /tmp/diskqueue-test/input.log
    prospector.scanner.fingerprint.enabled: false
    file_identity.native: ~

queue.disk:
  max_size: 100MB

output.console:
  pretty: true
```

```bash
cd filebeat && mage build
echo '{"message": "hello"}' > /tmp/diskqueue-test/input.log
./filebeat -e -c /path/to/filebeat-diskqueue-test.yml

# Verify: diskqueue created under path.data
ls -la /tmp/diskqueue-test/standalone/data/diskqueue/
# Should contain: state.dat (and .seg files once events flow)
```

### `elastic-agent otel`
Create `/tmp/diskqueue-test/otel-receivers.yml`:

```yaml
receivers:
  filebeatreceiver/a:
    filebeat:
      inputs:
        - type: filestream
          id: stream-a
          enabled: true
          paths:
            - /tmp/diskqueue-test/input.log
          prospector.scanner.fingerprint.enabled: false
          file_identity.native: ~
    path.home: /tmp/diskqueue-test/receiver-a
    queue.disk:
      max_size: 100MB

  filebeatreceiver/b:
    filebeat:
      inputs:
        - type: filestream
          id: stream-b
          enabled: true
          paths:
            - /tmp/diskqueue-test/input.log
          prospector.scanner.fingerprint.enabled: false
          file_identity.native: ~
    path.home: /tmp/diskqueue-test/receiver-b
    queue.disk:
      max_size: 100MB

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filebeatreceiver/a, filebeatreceiver/b]
      exporters: [debug]
  telemetry:
    metrics:
      level: none
```

Run:

```bash
elastic-agent otel --config /tmp/diskqueue-test/otel-receivers.yml
```

Verify:

```bash
find /tmp/diskqueue-test/receiver-a -name state.dat
# Expected: /tmp/diskqueue-test/receiver-a/data/diskqueue/state.dat

find /tmp/diskqueue-test/receiver-b -name state.dat
# Expected: /tmp/diskqueue-test/receiver-b/data/diskqueue/state.dat
```

## Related issues

- Closes #46989

(cherry picked from commit 979f9b4)

# Conflicts:
#	libbeat/outputs/logstash/logstash.go
#	x-pack/otel/exporter/logstashexporter/exporter.go
@mergify mergify bot requested a review from a team as a code owner March 4, 2026 15:10
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Mar 4, 2026
@mergify mergify bot requested review from leehinman and orestisfl and removed request for a team March 4, 2026 15:10
@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Mar 4, 2026

Cherry-pick of 979f9b4 has failed:

On branch mergify/bp/8.19/pr-48834
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 979f9b4a0.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   libbeat/cmd/instance/beat.go
	modified:   libbeat/cmd/test/output.go
	modified:   libbeat/outputs/console/console.go
	modified:   libbeat/outputs/discard/discard.go
	modified:   libbeat/outputs/elasticsearch/client_integration_test.go
	modified:   libbeat/outputs/elasticsearch/elasticsearch.go
	modified:   libbeat/outputs/fileout/file.go
	modified:   libbeat/outputs/kafka/client_test.go
	modified:   libbeat/outputs/kafka/kafka.go
	modified:   libbeat/outputs/kafka/kafka_integration_test.go
	modified:   libbeat/outputs/logstash/logstash_integration_test.go
	modified:   libbeat/outputs/logstash/logstash_test.go
	modified:   libbeat/outputs/output_reg.go
	modified:   libbeat/outputs/redis/redis.go
	modified:   libbeat/outputs/redis/redis_integration_test.go
	modified:   libbeat/outputs/redis/redis_test.go
	modified:   libbeat/outputs/util.go
	modified:   libbeat/outputs/util_test.go
	modified:   libbeat/publisher/pipeline/module.go
	modified:   libbeat/publisher/pipeline/pipeline.go
	modified:   libbeat/publisher/pipeline/stress/out.go
	modified:   libbeat/publisher/pipeline/stress/run.go
	modified:   libbeat/publisher/queue/diskqueue/benchmark_test.go
	modified:   libbeat/publisher/queue/diskqueue/config.go
	new file:   libbeat/publisher/queue/diskqueue/config_test.go
	modified:   libbeat/publisher/queue/diskqueue/core_loop.go
	modified:   libbeat/publisher/queue/diskqueue/core_loop_test.go
	modified:   libbeat/publisher/queue/diskqueue/deleter_loop.go
	modified:   libbeat/publisher/queue/diskqueue/queue.go
	modified:   libbeat/publisher/queue/diskqueue/queue_test.go
	modified:   libbeat/publisher/queue/diskqueue/reader_loop.go
	modified:   libbeat/publisher/queue/diskqueue/segments.go
	modified:   libbeat/publisher/queue/diskqueue/segments_test.go
	modified:   libbeat/publisher/queue/diskqueue/writer_loop.go
	modified:   x-pack/dockerlogbeat/pipelinemanager/libbeattools.go
	modified:   x-pack/filebeat/tests/integration/otel_test.go
	modified:   x-pack/libbeat/outputs/otelconsumer/otelconsumer.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   libbeat/outputs/logstash/logstash.go
	deleted by us:   x-pack/otel/exporter/logstashexporter/exporter.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 4, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 4, 2026

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@github-actions github-actions bot added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team skip-changelog labels Mar 4, 2026
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 4, 2026
@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Mar 9, 2026

This pull request has not been merged yet. Could you please review and merge it @orestisfl? 🙏

Resolve cherry-pick conflicts from #48834 backport to 8.19:
- logstash.go: keep 8.19 code structure (no MakeLogstashClients
  refactoring), add paths import and pass beatPaths to SuccessNet
- exporter.go: remove file that doesn't exist on 8.19 (the
  logstashexporter package is a main-branch feature)
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 9, 2026

The failure is in golangci-lint (run 22851947173, job lint (ubuntu-latest)), and the root cause is 8 linter violations reported in the golangci-lint step.

Summary of failing checks from the logs:

  • gosec (G115) (3):
    • libbeat/publisher/queue/diskqueue/segments.go:190 (int64 -> uint64)
    • libbeat/publisher/queue/diskqueue/segments.go:205 (int64 -> uint64)
    • libbeat/publisher/queue/diskqueue/writer_loop.go:197 (uint64 -> uint32)
  • noctx (2):
    • x-pack/filebeat/tests/integration/otel_test.go:1540
    • x-pack/filebeat/tests/integration/otel_test.go:1725
      (net.Listen should be replaced with net.ListenConfig{}.Listen(ctx, ...))
  • testifylint float-compare (3):
    • libbeat/outputs/logstash/logstash_test.go:153
    • x-pack/filebeat/tests/integration/otel_test.go:1109
    • x-pack/filebeat/tests/integration/otel_test.go:1113

Recommended minimal remediation:

  1. Add safe conversions/guards for the three numeric casts (bounds checks before casting).
  2. Replace both net.Listen(...) calls with (&net.ListenConfig{}).Listen(ctx, ...) using an explicit context.
  3. Replace assert.Equal float assertions with assert.InDelta (or InEpsilon) at appropriate tolerances.
  4. Re-run the same workflow/check after these targeted fixes.

Tests run by this investigation:

  • Reviewed failed GitHub Actions logs for job 66282406763 (no local test execution).

What is this? | From workflow: PR Actions Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 9, 2026

The failure is in golangci-lint (lint (ubuntu-latest)) and is caused by lint violations introduced in this backport, not infra flakiness.

What failed

Concrete failing locations from logs

  • libbeat/publisher/queue/diskqueue/segment_writer.goG115: integer overflow conversion uint64 -> uint32
  • libbeat/publisher/queue/diskqueue/writer_loop.go:197G115: integer overflow conversion uint64 -> uint32
  • x-pack/filebeat/tests/integration/otel_test.go:1540net.Listen must not be called (noctx)
  • x-pack/filebeat/tests/integration/otel_test.go:1725net.Listen must not be called (noctx)
  • libbeat/outputs/logstash/logstash_test.go:153float-compare: use assert.InEpsilon/InDelta
  • x-pack/filebeat/tests/integration/otel_test.go:1109float-compare
  • x-pack/filebeat/tests/integration/otel_test.go:1113float-compare

Minimal remediation plan

  1. Replace uint64 -> uint32 narrowing casts in diskqueue code with explicit bounds checks (or keep uint64 where possible) to satisfy gosec G115 safely.
  2. Replace net.Listen(...) with (&net.ListenConfig{}).Listen(ctx, ...) in otel_test.go.
  3. Replace exact float equality assertions with assert.InDelta/assert.InEpsilon in affected tests.

Tests run

  • I only analyzed GitHub Actions job metadata and failed-job logs for run 22851954807; no local test execution was performed in this workflow.

What is this? | From workflow: PR Actions Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

@orestisfl orestisfl merged commit cf7e66c into 8.19 Mar 11, 2026
205 of 208 checks passed
@orestisfl orestisfl deleted the mergify/bp/8.19/pr-48834 branch March 11, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport conflicts There is a conflict in the backported pull request skip-changelog Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant