Introduce Schema Registry Functionality to Managed KafkaIO Write.#35644
Introduce Schema Registry Functionality to Managed KafkaIO Write.#35644ahmedabu98 merged 12 commits intoapache:masterfrom
Conversation
8111242 to
260008c
Compare
|
Assigning reviewers: R: @m-trieu for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| implementation (group: 'com.google.cloud.hosted.kafka', name: 'managed-kafka-auth-login-handler', version: '1.0.5') { | ||
| // "kafka-clients" has to be provided since user can use its own version. | ||
| exclude group: 'org.apache.kafka', module: 'kafka-clients' | ||
| exclude group: "io.confluent", module: "kafka-schema-registry-client" |
There was a problem hiding this comment.
Can you please add a comment explaining why this exclusion is needed? Similar to kafka-clients above.
There was a problem hiding this comment.
added a comment. This is specified in the cloud documentation here https://cloud.google.com/managed-service-for-apache-kafka/docs/quickstart-avro#configure_and_run_the_producer but we want to use the schema registry client provided by the schema registry dependency instead of the transitive dependency here.
...a/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaWriteSchemaTransformProvider.java
Outdated
Show resolved
Hide resolved
260008c to
139f9fe
Compare
ahmedabu98
left a comment
There was a problem hiding this comment.
just some nits, but overall LGTM
would be great if we can include an integration test too
Allow users to write messages using a schema registry when utilizing ManagedIO for Google Dataflow. This will allow users to utilize the full capabilities of the Google Cloud Managed Service for Apache Kafka offering.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.