filebeat: stabilize kafka input integration topic readiness by github-actions[bot] · Pull Request #49228 · elastic/beats

github-actions · 2026-03-03T15:35:47Z

Proposed commit message

Use a shared helper that creates and waits for a test topic to become writable before producing events. Apply it across kafka integration tests, including TestTest, to reduce leader-election race flakes.

GenAI-Assisted: Yes
Human-Reviewed: Yes
Tool: GitHub Copilot CLI, Model: GPT-5.3 Codex

Summary

add createReadyTestTopic(t) to centralize kafka test-topic setup and readiness waiting
switch kafka input integration tests to use this helper before producing events
include TestTest and the other topic-writing tests so they all avoid leader-election write races consistently

Why

TestTest flake reports show transient no leader for this partition write errors. Other tests in the same file had the same write-on-fresh-topic pattern; this change applies the readiness guard uniformly.

Validation

go test -tags=integration ./filebeat/input/kafka -run '^(TestInput|TestInputWithMultipleEvents|TestInputWithJsonPayload|TestInputWithJsonPayloadAndMultipleEvents|TestSASLAuthentication|TestTest)$' -count=1 -v
- fails in this runner because Kafka test services are not running (connect: connection refused on localhost:9094), but the modified code path compiles and executes until environment connection.
Fixes [Flaky Test] Kafka TestTest: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes #49206

What is this? | From workflow: Mention in Issue

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Use a shared helper that creates and waits for a test topic to become writable before producing events. Apply it across kafka integration tests, including TestTest, to reduce leader-election race flakes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mergify · 2026-03-03T15:36:45Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @github-actions[bot]? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

belimawr · 2026-03-03T16:19:28Z

/ai create the changelog entry. If you need examples, look at the files in changelog/fragments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

elasticmachine · 2026-03-03T20:32:04Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

coderabbitai · 2026-03-03T20:34:45Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between dc2e6c4 and ce3b793.

📒 Files selected for processing (2)

changelog/fragments/1772555000-filebeat-kafka-topic-readiness.yaml
filebeat/input/kafka/kafka_integration_test.go

📝 Walkthrough

Walkthrough

Introduces a refactored Kafka topic creation pattern for Filebeat integration tests. Replaces the createTestTopicName() function with createReadyTestTopic(t), which encapsulates both topic creation and readiness validation. The new helper function invokes topic write readiness checks internally, eliminating scattered readiness calls across individual tests. These changes stabilize Kafka input integration tests by centralizing the logic that ensures topics are ready before testing, addressing intermittent leadership election failures. A changelog fragment documents the bug-fix.

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR directly addresses issue `#49206` by implementing a shared helper (createReadyTestTopic) that waits for Kafka topic write readiness before tests produce events, eliminating the leadership election race condition.
Out of Scope Changes check	✅ Passed	All changes are scoped to Kafka integration test stabilization: the changelog fragment, the new helper function, and updates to test calls. No unrelated modifications present.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings (stacked PR)
📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch fix/kafka-topic-readiness-tests-16054147cf2deff4

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

…readiness-tests-16054147cf2deff4

github-actions · 2026-03-03T21:14:14Z

CI failure is a golangci-lint timeout on macos-latest, not a lint violation. In run 22641594437, job lint (macos-latest) (65618809118) shows 0 issues followed by level=error msg="Timeout exceeded: try increasing it by passing --timeout option" and exits with code 4 (/tmp/gh-aw/agent/workflow-logs-22641594437/lint (macos-latest)/5_golangci-lint.txt:38-44).

Recommended fix: increase lint timeout in .github/workflows/golangci-lint.yml (args: --timeout=30m --whole-files at line 48), e.g. to --timeout=45m (or higher) to match worst-case macOS runtime; optional follow-up is reducing lint scope/runtime, but timeout bump is the minimal unblock.

Tests run for this investigation: log inspection only (no local code/tests executed).

What is this? | From workflow: PR Actions Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

github-actions · 2026-03-04T14:54:22Z

@Mergifyio backport 8.19 9.2 9.3

mergify · 2026-03-04T14:55:05Z

backport 8.19 9.2 9.3

✅ Backports have been created

Details

#49271 [8.19](backport #49228) filebeat: stabilize kafka input integration topic readiness has been created for branch 8.19
#49272 [9.2](backport #49228) filebeat: stabilize kafka input integration topic readiness has been created for branch 9.2
#49273 [9.3](backport #49228) filebeat: stabilize kafka input integration topic readiness has been created for branch 9.3

Use a shared helper that creates and waits for a test topic to become writable before producing events. Apply it across kafka integration tests, including TestTest, to reduce leader-election race flakes. GenAI-Assisted: Yes Human-Reviewed: Yes Tool: GitHub Copilot CLI, Model: GPT-5.3 Codex --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> (cherry picked from commit f6a7f39)

…49272) Use a shared helper that creates and waits for a test topic to become writable before producing events. Apply it across kafka integration tests, including TestTest, to reduce leader-election race flakes. GenAI-Assisted: Yes Human-Reviewed: Yes Tool: GitHub Copilot CLI, Model: GPT-5.3 Codex --------- (cherry picked from commit f6a7f39) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

…49273) Use a shared helper that creates and waits for a test topic to become writable before producing events. Apply it across kafka integration tests, including TestTest, to reduce leader-election race flakes. GenAI-Assisted: Yes Human-Reviewed: Yes Tool: GitHub Copilot CLI, Model: GPT-5.3 Codex --------- (cherry picked from commit f6a7f39) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

…49271) Use a shared helper that creates and waits for a test topic to become writable before producing events. Apply it across kafka integration tests, including TestTest, to reduce leader-election race flakes. GenAI-Assisted: Yes Human-Reviewed: Yes Tool: GitHub Copilot CLI, Model: GPT-5.3 Codex --------- (cherry picked from commit f6a7f39) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 3, 2026

Clean up code

bd85cd8

belimawr self-assigned this Mar 3, 2026

belimawr added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-active-all Automated backport with mergify to all the active branches labels Mar 3, 2026

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 3, 2026

Add changelog fragment for kafka topic readiness

ce3b793

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

belimawr marked this pull request as ready for review March 3, 2026 20:32

belimawr requested a review from a team as a code owner March 3, 2026 20:32

belimawr requested review from khushijain21 and mauri870 March 3, 2026 20:32

belimawr added the skip-changelog label Mar 3, 2026

belimawr added 2 commits March 3, 2026 15:35

Remove changelog

c874856

Merge branch 'main' of github.com:elastic/beats into fix/kafka-topic-…

e0dd1b0

…readiness-tests-16054147cf2deff4

belimawr approved these changes Mar 3, 2026

View reviewed changes

rdner approved these changes Mar 4, 2026

View reviewed changes

belimawr merged commit f6a7f39 into main Mar 4, 2026
48 of 49 checks passed

belimawr deleted the fix/kafka-topic-readiness-tests-16054147cf2deff4 branch March 4, 2026 14:54

mergify bot mentioned this pull request Mar 4, 2026

[8.19](backport #49228) filebeat: stabilize kafka input integration topic readiness #49271

Merged

mergify bot mentioned this pull request Mar 4, 2026

[9.2](backport #49228) filebeat: stabilize kafka input integration topic readiness #49272

Merged

mergify bot mentioned this pull request Mar 4, 2026

[9.3](backport #49228) filebeat: stabilize kafka input integration topic readiness #49273

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filebeat: stabilize kafka input integration topic readiness#49228

filebeat: stabilize kafka input integration topic readiness#49228
belimawr merged 5 commits intomainfrom
fix/kafka-topic-readiness-tests-16054147cf2deff4

github-actions bot commented Mar 3, 2026 •

edited by belimawr

Loading

Uh oh!

mergify bot commented Mar 3, 2026

Uh oh!

belimawr commented Mar 3, 2026

Uh oh!

elasticmachine commented Mar 3, 2026

Uh oh!

coderabbitai bot commented Mar 3, 2026

Walkthrough

Uh oh!

github-actions bot commented Mar 3, 2026

Uh oh!

Uh oh!

github-actions bot commented Mar 4, 2026

Uh oh!

mergify bot commented Mar 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

github-actions bot commented Mar 3, 2026 • edited by belimawr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Summary

Why

Validation

Uh oh!

mergify bot commented Mar 3, 2026

Uh oh!

belimawr commented Mar 3, 2026

Uh oh!

elasticmachine commented Mar 3, 2026

Uh oh!

coderabbitai bot commented Mar 3, 2026

Walkthrough

Uh oh!

github-actions bot commented Mar 3, 2026

Uh oh!

Uh oh!

github-actions bot commented Mar 4, 2026

Uh oh!

mergify bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Mar 3, 2026 •

edited by belimawr

Loading

mergify bot commented Mar 4, 2026 •

edited

Loading