[beatreceiver] - Disable `force_attempt_http2` and map ES config to elasticsearchexporter by VihasMakwana · Pull Request #46111 · elastic/beats

VihasMakwana · 2025-08-20T10:54:30Z

Proposed commit message

This PR fixes the presets for otel mode by disabling force_attempt_http2 and mapping the libbeat's workers to upstream's max_conns_per_hosts setting.
It also updates go.mod to latest collector release.

It also moves the batcher config to sending_queue as the previous way is deprecated.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

How to test this PR locally

To test the total number of connections to the ES host opened by the process, you can follow these steps:
1. Run filebeat in otel mode
2. Take a note of PID
3. In a separate terminal, do lsof -iTCP -nP | grep pid and it should show you the the number of open tcp connections.
  - For throughput preset, you should see 4 connections open
  - For other presets, only single connection should be open.
Run filebeat otel inspect command and it will output following config:

exporters:
    elasticsearch:
        compression: gzip
        compression_params:
            level: 1
        endpoints:
            - https://my-deployment-3ccdf5.es.us-central1.gcp.cloud.es.io:443
        force_attempt_http2: false
        idle_conn_timeout: 15s
        mapping:
            mode: bodymap
        max_conns_per_host: 4
        password: ...
        retry:
            enabled: true
            initial_interval: 1s
            max_interval: 1m0s
            max_retries: 3
        sending_queue:
            batch:
                max_size: 1600
                min_size: 0
                sizer: items
            block_on_overflow: true
            enabled: true
            num_consumers: 4
            queue_size: 9223372036854775807
            wait_for_result: true
        timeout: 1m30s
        user: elastic
receivers:
    filebeatreceiver:
        filebeat:
            inputs:
                - enabled: true
                  id: filestream-input-id
                  paths:
                    - /Users/vihasmakwana/Desktop/Vihas/elastic/ingest-dev/performance/use_cases/bb/benchmark/apache-logs-0.ndjson
                  type: filestream
        http:
            enabled: true
            pprof:
                enabled: true
        output:
            otelconsumer: null
        path:
            config: .
            data: ./data
            home: .
            logs: ./logs
        queue:
            mem:
                events: 12800
                flush:
                    min_events: 1600
                    timeout: 5s
service:
    pipelines:
        logs:
            exporters:
                - elasticsearch
            receivers:
                - filebeatreceiver

Related issues

Relates https://github.com/elastic/ingest-dev/issues/5823

elasticmachine · 2025-08-20T10:54:34Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

github-actions · 2025-08-20T10:54:38Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2025-08-20T10:55:17Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @VihasMakwana? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

mergify · 2025-08-29T06:39:31Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b presets upstream/presets
git merge upstream/main
git push upstream presets

cmacknz · 2025-09-03T15:27:09Z

Added the backport 8.19 label as we should backport beats receivers changes to 8.19 to keep the code structure the same, and allow us to make the switch to beat receivers once we feel it's ready (sometime after it has been stable in 9.x).

mauri870 · 2025-09-05T14:04:56Z

@VihasMakwana I just want to point out that I have a test I worked on for delivery guarantees and document-level failures that relies on the sync batcher option. I just tested it with this PR, since it migrated to sending_queue, and I can confirm the same delivery guarantees still hold. It’s more of a proto-test, but it’s reassuring to know this didn’t regress any of the behavior we previously built upon with the batcher.

mergify · 2025-09-08T12:04:00Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b presets upstream/presets
git merge upstream/main
git push upstream presets

github-actions · 2025-09-11T05:12:19Z

@Mergifyio backport 9.0 9.1

mergify · 2025-09-11T05:12:33Z

backport 9.0 9.1

✅ Backports have been created

Details

#46532 [9.0](backport #46111) [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter has been created for branch 9.0 but encountered conflicts
#46533 [9.1](backport #46111) [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter has been created for branch 9.1 but encountered conflicts

…lasticsearchexporter (#46111) * initial commit * minor change * comments and go.mod * go.mod * comments and notice * tests * remove maxInt * chore: fix tests and add batcher in sending_queue * beatconverter tests * detailed comment * omit force_attempt_http2 for now (cherry picked from commit 02b596f) # Conflicts: # NOTICE.txt # go.mod # go.sum # libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go # libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel_test.go # x-pack/filebeat/tests/integration/otel_test.go # x-pack/metricbeat/tests/integration/otel_test.go

…lasticsearchexporter (#46111) * initial commit * minor change * comments and go.mod * go.mod * comments and notice * tests * remove maxInt * chore: fix tests and add batcher in sending_queue * beatconverter tests * detailed comment * omit force_attempt_http2 for now (cherry picked from commit 02b596f) # Conflicts: # NOTICE.txt # go.mod # go.sum

…lasticsearchexporter (#46111) * initial commit * minor change * comments and go.mod * go.mod * comments and notice * tests * remove maxInt * chore: fix tests and add batcher in sending_queue * beatconverter tests * detailed comment * omit force_attempt_http2 for now (cherry picked from commit 02b596f) # Conflicts: # NOTICE.txt # go.mod # go.sum # x-pack/filebeat/tests/integration/otel_test.go # x-pack/metricbeat/tests/integration/otel_test.go

…lasticsearchexporter (#46111) * initial commit * minor change * comments and go.mod * go.mod * comments and notice * tests * remove maxInt * chore: fix tests and add batcher in sending_queue * beatconverter tests * detailed comment * omit force_attempt_http2 for now

… and map ES config to elasticsearchexporter (#46533) * [beatreceiver] - Disable `force_attempt_http2` and map ES config to elasticsearchexporter (#46111) * initial commit * minor change * comments and go.mod * go.mod * comments and notice * tests * remove maxInt * chore: fix tests and add batcher in sending_queue * beatconverter tests * detailed comment * omit force_attempt_http2 for now (cherry picked from commit 02b596f) # Conflicts: # NOTICE.txt # go.mod # go.sum * merge conflicts --------- Co-authored-by: Vihas Makwana <121151420+VihasMakwana@users.noreply.github.com> Co-authored-by: Vihas <vihas.makwana@elastic.co>

…` and map ES config to elasticsearchexporter (#46534) * [beatreceiver] - Disable `force_attempt_http2` and map ES config to elasticsearchexporter (#46111) * initial commit * minor change * comments and go.mod * go.mod * comments and notice * tests * remove maxInt * chore: fix tests and add batcher in sending_queue * beatconverter tests * detailed comment * omit force_attempt_http2 for now (cherry picked from commit 02b596f) # Conflicts: # NOTICE.txt # go.mod # go.sum # x-pack/filebeat/tests/integration/otel_test.go # x-pack/metricbeat/tests/integration/otel_test.go * merge conflicts * merge conflicts --------- Co-authored-by: Vihas Makwana <121151420+VihasMakwana@users.noreply.github.com> Co-authored-by: Vihas <vihas.makwana@elastic.co>

initial commit

cf9faec

VihasMakwana self-assigned this Aug 20, 2025

VihasMakwana requested a review from a team as a code owner August 20, 2025 10:54

VihasMakwana added the bug label Aug 20, 2025

VihasMakwana requested a review from a team as a code owner August 20, 2025 10:54

VihasMakwana added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Aug 20, 2025

VihasMakwana requested review from andrzej-stencel and rdner August 20, 2025 10:54

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 20, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 20, 2025

VihasMakwana marked this pull request as draft August 20, 2025 10:54

VihasMakwana changed the title ~~[beatreceiver] - Disable force_attempt_http2 and map workers * num_hosts to max_conns_per_hosts~~ [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter Aug 20, 2025

VihasMakwana and others added 5 commits August 21, 2025 13:27

minor change

0d5f20b

comments and go.mod

ad7f5c5

go.mod

9140e13

comments and notice

342a1da

Merge branch 'main' into presets

5071302

VihasMakwana marked this pull request as ready for review August 25, 2025 10:16

VihasMakwana requested review from leehinman and removed request for rdner August 25, 2025 10:16

VihasMakwana commented Aug 25, 2025

View reviewed changes

Comment thread libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go

VihasMakwana commented Aug 25, 2025

View reviewed changes

Comment thread libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go

VihasMakwana added the backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches label Aug 25, 2025

tests

3a7c251

leehinman reviewed Aug 25, 2025

View reviewed changes

Comment thread libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go Outdated

Merge branch 'main' into presets

fe6fb6b

cmacknz added the backport-8.19 Automated backport to the 8.19 branch label Sep 3, 2025

cmacknz reviewed Sep 3, 2025

View reviewed changes

Comment thread libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go

cmacknz reviewed Sep 3, 2025

View reviewed changes

Comment thread libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go

cmacknz approved these changes Sep 3, 2025

View reviewed changes

detailed comment

db985d6

andrzej-stencel approved these changes Sep 4, 2025

View reviewed changes

mauri870 mentioned this pull request Sep 5, 2025

otel: add test for otelconsumer + elasticsearchexporter document-level retries #46413

Merged

6 tasks

cmacknz reviewed Sep 9, 2025

View reviewed changes

Comment thread libbeat/otelbeat/beatconverter/beatconverter_test.go Outdated

cmacknz mentioned this pull request Sep 10, 2025

[beatreceivers] Integrate beatsauthextension elastic/elastic-agent#9257

Merged

8 tasks

VihasMakwana added 2 commits September 11, 2025 00:50

Merge branch 'main' into presets

a3d875f

omit force_attempt_http2 for now

e518923

VihasMakwana merged commit 02b596f into elastic:main Sep 11, 2025
204 of 207 checks passed

mergify bot mentioned this pull request Sep 11, 2025

[9.0](backport #46111) [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter #46532

Closed

6 tasks

This was referenced Sep 11, 2025

[9.1](backport #46111) [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter #46533

Merged

[8.19](backport #46111) [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter #46534

Merged

mauri870 mentioned this pull request Sep 12, 2025

[main][Automation] Update elastic/beats to afc53c0479ac elastic/elastic-agent#9874

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[beatreceiver] - Disable `force_attempt_http2` and map ES config to elasticsearchexporter#46111

[beatreceiver] - Disable `force_attempt_http2` and map ES config to elasticsearchexporter#46111
VihasMakwana merged 15 commits intoelastic:mainfrom
VihasMakwana:presets

VihasMakwana commented Aug 20, 2025 •

edited

Loading

Uh oh!

elasticmachine commented Aug 20, 2025

Uh oh!

github-actions bot commented Aug 20, 2025

Uh oh!

mergify bot commented Aug 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Aug 29, 2025

Uh oh!

Uh oh!

Uh oh!

cmacknz commented Sep 3, 2025

Uh oh!

mauri870 commented Sep 5, 2025

Uh oh!

mergify bot commented Sep 8, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 11, 2025

Uh oh!

mergify bot commented Sep 11, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

VihasMakwana commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Aug 20, 2025

Uh oh!

github-actions bot commented Aug 20, 2025

🤖 GitHub comments

Uh oh!

mergify bot commented Aug 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Aug 29, 2025

Uh oh!

Uh oh!

Uh oh!

cmacknz commented Sep 3, 2025

Uh oh!

mauri870 commented Sep 5, 2025

Uh oh!

mergify bot commented Sep 8, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 11, 2025

Uh oh!

mergify bot commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

VihasMakwana commented Aug 20, 2025 •

edited

Loading

mergify bot commented Sep 11, 2025 •

edited

Loading