Fix Kafka flaky tests by rdner · Pull Request #48785 · elastic/beats

rdner · 2026-02-10T19:20:32Z

Proposed commit message

Added delays and waits at certain steps to ensure the Kafka configuration running properly before starting the test.
Add connection retries for the Kafka producer

Assisted by Cursor.

The Kafka tests are flaky and sometime fails with:

=== FAIL: filebeat/input/kafka TestSASLAuthentication/PLAIN (0.93s)
kafka_integration_test.go:369: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes

For example, in this build https://buildkite.com/elastic/filebeat/builds/28161#019c482f-481b-40db-85e6-34da7056207f

Relates to #48026

github-actions · 2026-02-10T19:20:46Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

* Added delays and waits at certain steps to ensure the Kafka configuration running properly before starting the test. * Add connection retries for the Kafka producer Assisted by Cursor.

elasticmachine · 2026-02-10T20:08:38Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

rdner · 2026-02-10T20:22:29Z

Looks like it still fails:

=== FAIL: filebeat/input/kafka TestTest (1.03s)
kafka_integration_test.go:462: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes

I'm taking this back to draft and will investigate.

rdner · 2026-02-10T20:43:40Z

@faec is it a requirement to have 3 partitions for Kafka? I cannot find any test that requires this:

https://github.com/elastic/beats/blame/b34256a57d1a1d441a2787bb55302434d4978c94/testing/environments/docker/kafka/run.sh#L37

filebeat/input/kafka/kafka_integration_test.go

rdner · 2026-02-11T08:57:57Z

I'm going to run the tests on this PR multiple times to make sure it actually fixed the flaky behavior. After this confirmation the PR can be merged.

rdner · 2026-02-11T09:59:00Z

Failed again with:

kafka_integration_test.go:463: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes

I'll investigate further but the easiest way to fix this would be switching to a single partition instead of 3.

testing/environments/docker/kafka/run.sh

…tests

Create the test topic explicitly and wait for partition leaders before producing messages, preventing transient CI failures with “no leader for this partition” during topic auto-creation. GenAI-Assisted: Yes Human-Reviewed: Yes Tool: Cursor CLI, Model: GPT-5.3 Codex

rdner · 2026-02-25T18:02:41Z

/test

rdner · 2026-02-26T09:00:27Z

Looks like 3 consecutive CI runs didn't fail. Let's merge it and observe.
@belimawr if you share my confidence, please approve the PR 🙂

belimawr · 2026-02-26T14:00:07Z

@belimawr if you share my confidence, please approve the PR 🙂

Yes I share the confidence!

Also, the test is already flaky: best case scenario the flakiness is fixed, worst case it stays flaky and we learn that this was not the root cause. So, merging it it's a win-win.

github-actions · 2026-02-26T14:03:02Z

@Mergifyio backport 8.19 9.2 9.3

mergify · 2026-02-26T14:03:12Z

backport 8.19 9.2 9.3

✅ Backports have been created

Details

#49140 [8.19](backport #48785) Fix Kafka flaky tests has been created for branch 8.19
#49141 [9.2](backport #48785) Fix Kafka flaky tests has been created for branch 9.2
#49142 [9.3](backport #48785) Fix Kafka flaky tests has been created for branch 9.3

* Added delays and waits at certain steps to ensure the Kafka configuration running properly before starting the test. * Add connection retries for the Kafka producer Assisted by Cursor. * Try to disable leader election * Replace the attempts loop with the eventually call * Make the helper more debuggable * filebeat/input/kafka: stabilize TestSASLAuthentication topic setup Create the test topic explicitly and wait for partition leaders before producing messages, preventing transient CI failures with “no leader for this partition” during topic auto-creation. GenAI-Assisted: Yes Human-Reviewed: Yes Tool: Cursor CLI, Model: GPT-5.3 Codex --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> (cherry picked from commit 0132a25)

* Added delays and waits at certain steps to ensure the Kafka configuration running properly before starting the test. * Add connection retries for the Kafka producer * Create the test topic explicitly and wait for partition leaders before producing messages, preventing transient CI failures with “no leader for this partition” during topic auto-creation. Assisted by Cursor. --------- (cherry picked from commit 0132a25) Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

belimawr · 2026-03-02T21:57:28Z

I failed again in a backport PR to 9.2, even after the fix was merged.

Stack trace:

=== Failed
=== FAIL: filebeat/input/kafka TestTest (1.07s)
    kafka_integration_test.go:676:
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1772484810880252862/elastic/filebeat/filebeat/input/kafka/kafka_integration_test.go:676
        	            				/opt/buildkite-agent/builds/bk-agent-prod-gcp-1772484810880252862/elastic/filebeat/filebeat/input/kafka/kafka_integration_test.go:464
        	Error:      	Received unexpected error:
        	            	kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes
        	Test:       	TestTest

I'll create a flaky test issue for this.

rdner · 2026-03-03T08:14:33Z

@belimawr thanks for creating the issue and investigating!

It might be an environmental problem too. Perhaps some buildkite runners have something different about them.

belimawr · 2026-03-03T15:32:53Z

@belimawr thanks for creating the issue and investigating!

It might be an environmental problem too. Perhaps some buildkite runners have something different about them.

I was looking more into it, well I mostly delegated the analysis to AI. Even though it is the same error, the failure is in another test, same root cause.

We'll (AI and I) fix it in all test for this input.

rdner self-assigned this Feb 10, 2026

rdner added Filebeat Filebeat flaky-test Unstable or unreliable test cases. Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Feb 10, 2026

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 10, 2026

rdner added needs_team Indicates that the issue/PR needs a Team:* label backport-active-all Automated backport with mergify to all the active branches labels Feb 10, 2026

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 10, 2026

rdner added the skip-changelog label Feb 10, 2026

rdner requested review from khushijain21 and mauri870 February 10, 2026 19:30

Fix Kafka flaky tests

722b13f

* Added delays and waits at certain steps to ensure the Kafka configuration running properly before starting the test. * Add connection retries for the Kafka producer Assisted by Cursor.

rdner force-pushed the fix-flaky-kafka-tests branch from b7f8b79 to 722b13f Compare February 10, 2026 19:34

rdner mentioned this pull request Feb 10, 2026

[9.2](backport #48749) chore: replace deprecated docker-compose with docker compose #48769

Merged

rdner marked this pull request as ready for review February 10, 2026 20:08

rdner requested a review from a team as a code owner February 10, 2026 20:08

rdner marked this pull request as draft February 10, 2026 20:22

Try to disable leader election

7ade053

VihasMakwana reviewed Feb 11, 2026

View reviewed changes

filebeat/input/kafka/kafka_integration_test.go Outdated Show resolved Hide resolved

Replace the attempts loop with the eventually call

97ad8f8

rdner requested a review from faec February 11, 2026 12:31

mauri870 reviewed Feb 11, 2026

View reviewed changes

testing/environments/docker/kafka/run.sh Show resolved Hide resolved

Make the helper more debuggable

2022bef

belimawr reviewed Feb 24, 2026

View reviewed changes

testing/environments/docker/kafka/run.sh Outdated Show resolved Hide resolved

belimawr added 2 commits February 24, 2026 09:46

Merge branch 'main' of github.com:elastic/beats into fix-flaky-kafka-…

f233d9a

…tests

rdner marked this pull request as ready for review February 25, 2026 13:59

rdner requested review from VihasMakwana and mauri870 February 25, 2026 13:59

Merge branch 'main' into fix-flaky-kafka-tests

e558055

orestisfl approved these changes Feb 26, 2026

View reviewed changes

mauri870 approved these changes Feb 26, 2026

View reviewed changes

belimawr approved these changes Feb 26, 2026

View reviewed changes

belimawr merged commit 0132a25 into elastic:main Feb 26, 2026
191 checks passed

mergify bot mentioned this pull request Feb 26, 2026

[8.19](backport #48785) Fix Kafka flaky tests #49140

Merged

mergify bot mentioned this pull request Feb 26, 2026

[9.2](backport #48785) Fix Kafka flaky tests #49141

Merged

mergify bot mentioned this pull request Feb 26, 2026

[9.3](backport #48785) Fix Kafka flaky tests #49142

Merged

rdner deleted the fix-flaky-kafka-tests branch February 26, 2026 14:08

belimawr mentioned this pull request Mar 2, 2026

[Flaky Test] Kafka TestTest: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes #49206

Closed

Conversation

rdner commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Uh oh!

github-actions bot commented Feb 10, 2026

🤖 GitHub comments

Uh oh!

elasticmachine commented Feb 10, 2026

Uh oh!

rdner commented Feb 10, 2026

Uh oh!

rdner commented Feb 10, 2026

Uh oh!

Uh oh!

rdner commented Feb 11, 2026

Uh oh!

rdner commented Feb 11, 2026

Uh oh!

Uh oh!

Uh oh!

rdner commented Feb 25, 2026

Uh oh!

rdner commented Feb 26, 2026

Uh oh!

belimawr commented Feb 26, 2026

Uh oh!

Uh oh!

github-actions bot commented Feb 26, 2026

Uh oh!

mergify bot commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

belimawr commented Mar 2, 2026

Uh oh!

rdner commented Mar 3, 2026

Uh oh!

belimawr commented Mar 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rdner commented Feb 10, 2026 •

edited

Loading

mergify bot commented Feb 26, 2026 •

edited

Loading