This repository was archived by the owner on Sep 21, 2023. It is now read-only.
Add support for topic "selectors" to Kafka output#199
Merged
robbavey merged 8 commits intoelastic:mainfrom Dec 13, 2022
Merged
Conversation
A subsequent call to batch.Done will cause the read loop to fail
Contributor
|
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
Contributor
faec
approved these changes
Dec 12, 2022
Contributor
faec
left a comment
There was a problem hiding this comment.
Thanks! This looks good as a stopgap but it makes me nervous that it expands our dependence on beat.Event, which we ultimately want to remove from the shipper. Can you open a followup issue to add "real" selector support to the Kafka output that doesn't depend on the old libbeat data structures?
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This commit adds support for topic selectors to provide parity with the existing beats Kafka output - see https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#topic-option-kafka and https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#topics-option-kafka for details on how to use this function.
This PR continues to use the
outil.Selectorfromlibbeatas used in the beats Kafka outputChecklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added an entry inCHANGELOG.mdorCHANGELOG-developer.md.How to test this PR locally
Testing can be done the same way as #147, with changes for the kafka output:
Run a local Kafka running on port 9092, and using metricbeat with:
And the following settings in elastic-agent-shipper.yml
And using kcat to pull messages out of the various topics to prove the correctness of delivery.
Related issues
Closes #167