Skip to content

[libbeat] kafka message support headers feature#29940

Merged
rdner merged 6 commits intoelastic:mainfrom
herbxu:kafka-output-add-headers
Feb 12, 2022
Merged

[libbeat] kafka message support headers feature#29940
rdner merged 6 commits intoelastic:mainfrom
herbxu:kafka-output-add-headers

Conversation

@herbxu
Copy link
Copy Markdown
Contributor

@herbxu herbxu commented Jan 21, 2022 •
edited
Loading

What does this PR do?

kafka message support headers feature

Why is it important?

Kafka Message Header lets consumers process message metadata without having to deserialize the message value/payload. This could be useful in the routing of message use case.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

xiangwei.xxw@alibaba-inc.com

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 21, 2022
@cla-checker-service
Copy link
Copy Markdown

cla-checker-service bot commented Jan 21, 2022 •
edited
Loading

💚 CLA has been signed

@herbxu herbxu changed the title kafka message support headers feature [libbeat] kafka message support headers feature Jan 21, 2022
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jan 21, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-02-11T22:30:06.048+0000

  • Duration: 60 min 9 sec

Test stats 🧪

Test Results
Failed 0
Passed 28215
Skipped 3032
Total 31247

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Jan 21, 2022

This pull request does not have a backport label. Could you fix it @herbxu? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Jan 21, 2022
@mtojek mtojek added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Jan 24, 2022
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jan 24, 2022

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 24, 2022
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Feb 1, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kafka-output-add-headers upstream/kafka-output-add-headers
git merge upstream/master
git push upstream kafka-output-add-headers

@mergify mergify bot assigned herbxu Feb 8, 2022
@rdner rdner self-requested a review February 9, 2022 15:20
rdner
rdner reviewed Feb 9, 2022
View reviewed changes
rdner
rdner reviewed Feb 9, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@rdner rdner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution, @herbxu!

In order to merge this PR, we would need a test confirming the new functionality works (see https://github.com/elastic/beats/blob/main/libbeat/outputs/kafka/kafka_integration_test.go). Once it's there, I'm happy to merge the PR.

@herbxu
Copy link
Copy Markdown
Contributor Author

herbxu commented Feb 10, 2022

Thank you for your contribution, @herbxu!

In order to merge this PR, we would need a test confirming the new functionality works (see https://github.com/elastic/beats/blob/main/libbeat/outputs/kafka/kafka_integration_test.go). Once it's there, I'm happy to merge the PR.

I have added the integration test for this feature, thanks !

@herbxu herbxu requested a review from rdner February 11, 2022 01:59
rdner
rdner reviewed Feb 11, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@rdner rdner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for addressing the comments and adding the test case. I noticed another small improvement that you might want to apply.

It looks like you have not signed the CLA https://www.elastic.co/contributor-agreement

Since you use the privacy feature on GitHub and your commits have a different email address, the automation is not able to detect the CLA signature. However, I checked manually in our contributor database and I could find neither your nickname herbxu nor your email address xiangwei.xxw@alibaba-inc.com

After the CLA is signed I'll run the CI pipeline and we can merge this PR.

@herbxu @rdner
Update libbeat/outputs/kafka/client.go
e90c2c1 
Accepted this pre-allocate memory for kafka headers buffer

Co-authored-by: Denis Rechkunov <denis@rdner.de>
@herbxu
Copy link
Copy Markdown
Contributor Author

herbxu commented Feb 11, 2022

Thank you for addressing the comments and adding the test case. I noticed another small improvement that you might want to apply.

It looks like you have not signed the CLA https://www.elastic.co/contributor-agreement

Since you use the privacy feature on GitHub and your commits have a different email address, the automation is not able to detect the CLA signature. However, I checked manually in our contributor database and I could find neither your nickname herbxu nor your email address xiangwei.xxw@alibaba-inc.com

After the CLA is signed I'll run the CI pipeline and we can merge this PR.

I have signed with the email xiangwei.xxw@alibaba-inc.com, thanks for your suggestion.

@rdner rdner added the v8.2.0 label Feb 11, 2022
@rdner
Copy link
Copy Markdown
Member

rdner commented Feb 11, 2022 •
edited
Loading

jenkins run tests

@rdner rdner added the libbeat label Feb 11, 2022
@rdner
Copy link
Copy Markdown
Member

rdner commented Feb 11, 2022

/test

@rdner rdner merged commit 1bba206 into elastic:main Feb 12, 2022
@rdner rdner mentioned this pull request Feb 12, 2022
Merged
5 tasks
@rdner
Copy link
Copy Markdown
Member

rdner commented Feb 14, 2022

@herbxu just FYI, I had to modify the configuration structure a little bit, see #30369

v1v added a commit to v1v/beats that referenced this pull request Feb 21, 2022
@v1v
Merge branch 'feature/use-with-kind-k8s-env' of github.com:v1v/beats …
c20f180 
…into feature/use-with-kind-k8s-env

* 'feature/use-with-kind-k8s-env' of github.com:v1v/beats: (52 commits)
  ci: home is declared within withBeatsEnv
  ci: use withKindEnv step
  ci: use getBranchesFromAliases and support next-patch-8 (elastic#30400)
  Update fields.yml (elastic#29609)
  Heartbeat: fix browser metrics and trace mappings (elastic#30258)
  Apply light edits to 8.0 changelog (elastic#30351)
  packetbeat/beater: make sure Npcap installation runs before interfaces are needed (elastic#30396)
  Add a ring-buffer reporter to libbeat (elastic#28750)
  Osquerybeat: Add install verification for osquerybeat (elastic#30388)
  update windows matrix support (elastic#30373)
  Refactor of metricbeat process-gathering metrics and system/process (elastic#30076)
  adjust next changelog wording (elastic#30371)
  [Metricbeat] azure: move event report into loop validDim loop (elastic#29945)
  fix: report GitHub Check before the cache (elastic#30372)
  Add support for non-unique keys in Kafka output headers (elastic#30369)
  ci: 6 major branch reached EOL (elastic#30357)
  reduce Elastic Agent shut down time by stopping processes concurrently (elastic#29650)
  [Filebeat] Add message to register encode/decode debug logs (elastic#30271)
  [libbeat] kafka message header support (elastic#29940)
  Heartbeat: set duration to zero for syntax errors (elastic#30227)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rdner rdner rdner approved these changes

Assignees

@herbxu herbxu

Labels

backport-skip Skip notification from the automated backport with mergify libbeat Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team v8.2.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@herbxu @elasticmachine @rdner @mtojek @jlind23