Skip to content

[9.0](backport #46111) [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter#46532

Closed
mergify[bot] wants to merge 2 commits into9.0from
mergify/bp/9.0/pr-46111
Closed

[9.0](backport #46111) [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter#46532
mergify[bot] wants to merge 2 commits into9.0from
mergify/bp/9.0/pr-46111

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Sep 11, 2025

Proposed commit message

This PR fixes the presets for otel mode by disabling force_attempt_http2 and mapping the libbeat's workers to upstream's max_conns_per_hosts setting.
It also updates go.mod to latest collector release.

It also moves the batcher config to sending_queue as the previous way is deprecated.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

How to test this PR locally

  • To test the total number of connections to the ES host opened by the process, you can follow these steps:

    1. Run filebeat in otel mode
    2. Take a note of PID
    3. In a separate terminal, do lsof -iTCP -nP | grep pid and it should show you the the number of open tcp connections.
      • For throughput preset, you should see 4 connections open
      • For other presets, only single connection should be open.

  • Run filebeat otel inspect command and it will output following config:

exporters:
    elasticsearch:
        compression: gzip
        compression_params:
            level: 1
        endpoints:
            - https://my-deployment-3ccdf5.es.us-central1.gcp.cloud.es.io:443
        force_attempt_http2: false
        idle_conn_timeout: 15s
        mapping:
            mode: bodymap
        max_conns_per_host: 4
        password: ...
        retry:
            enabled: true
            initial_interval: 1s
            max_interval: 1m0s
            max_retries: 3
        sending_queue:
            batch:
                max_size: 1600
                min_size: 0
                sizer: items
            block_on_overflow: true
            enabled: true
            num_consumers: 4
            queue_size: 9223372036854775807
            wait_for_result: true
        timeout: 1m30s
        user: elastic
receivers:
    filebeatreceiver:
        filebeat:
            inputs:
                - enabled: true
                  id: filestream-input-id
                  paths:
                    - /Users/vihasmakwana/Desktop/Vihas/elastic/ingest-dev/performance/use_cases/bb/benchmark/apache-logs-0.ndjson
                  type: filestream
        http:
            enabled: true
            pprof:
                enabled: true
        output:
            otelconsumer: null
        path:
            config: .
            data: ./data
            home: .
            logs: ./logs
        queue:
            mem:
                events: 12800
                flush:
                    min_events: 1600
                    timeout: 5s
service:
    pipelines:
        logs:
            exporters:
                - elasticsearch
            receivers:
                - filebeatreceiver

Related issues

This is an automatic backport of pull request #46111 done by [Mergify](https://mergify.com).

@VihasMakwana @mergify
[beatreceiver] - Disable force_attempt_http2 and map ES config to e…
38f208e 
…lasticsearchexporter (#46111)

* initial commit

* minor change

* comments and go.mod

* go.mod

* comments and notice

* tests

* remove maxInt

* chore: fix tests and add batcher in sending_queue

* beatconverter tests

* detailed comment

* omit force_attempt_http2 for now

(cherry picked from commit 02b596f)

# Conflicts:
#	NOTICE.txt
#	go.mod
#	go.sum
#	libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go
#	libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel_test.go
#	x-pack/filebeat/tests/integration/otel_test.go
#	x-pack/metricbeat/tests/integration/otel_test.go
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Sep 11, 2025
@mergify mergify bot requested review from a team as code owners September 11, 2025 05:12
@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Sep 11, 2025

Cherry-pick of 02b596f has failed:

On branch mergify/bp/9.0/pr-46111
Your branch is up to date with 'origin/9.0'.

You are currently cherry-picking commit 02b596f4a.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   libbeat/otelbeat/beatconverter/beatconverter_test.go
	modified:   libbeat/publisher/queue/memqueue/config.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   NOTICE.txt
	both modified:   go.mod
	both modified:   go.sum
	both modified:   libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go
	both modified:   libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel_test.go
	both modified:   x-pack/filebeat/tests/integration/otel_test.go
	deleted by us:   x-pack/metricbeat/tests/integration/otel_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@mergify mergify bot added the backport label Sep 11, 2025
@mergify mergify bot added the conflicts There is a conflict in the backported pull request label Sep 11, 2025
@mergify mergify bot requested review from AndersonQ and VihasMakwana and removed request for a team September 11, 2025 05:12
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 11, 2025
@mergify mergify bot mentioned this pull request Sep 11, 2025
Merged
6 tasks
@github-actions github-actions bot added bug Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Sep 11, 2025
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 11, 2025

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 11, 2025
@mergify mergify bot deleted the mergify/bp/9.0/pr-46111 branch September 11, 2025 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@VihasMakwana VihasMakwana VihasMakwana approved these changes

@AndersonQ AndersonQ Awaiting requested review from AndersonQ AndersonQ is a code owner automatically assigned from elastic/elastic-agent-data-plane

Assignees

@VihasMakwana VihasMakwana

Labels

backport bug conflicts There is a conflict in the backported pull request Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@elasticmachine @VihasMakwana