Add Error Handling to Kafka IO#29546
Merged
johnjcasey merged 105 commits intoapache:masterfrom Dec 7, 2023
Merged
Conversation
# Conflicts: # CHANGES.md
Create gradle task and github actions config for GCS using this.
…tests Feature/automate performance tests
…tests fix call to gradle
…tests run on hosted runner for testing
…tests add additional checkout
…tests add destination for triggered tests
…tests move env variables to correct location
…tests try uploading against separate dataset
…tests try without a user
…tests update branch checkout, try to view the failure log
…tests run on failure
…tests update to use correct BigQuery instance
…tests convert to matrix
update error handler to be serializable to support using it as a member of an auto-value based PTransform
…er-queue-core # Conflicts: # CHANGES.md
…d-dlq # Conflicts: # sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/errorhandling/ErrorHandler.java
Contributor
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
shunping
reviewed
Dec 6, 2023
| project.ext { | ||
| delimited="0.11.0.3" | ||
| undelimited="01103" | ||
| sdfCompatable=false |
Collaborator
There was a problem hiding this comment.
is "Compatable" a typo here?
shunping
reviewed
Dec 6, 2023
| abstract Builder<K, V> setCheckStopReadingFn(@Nullable CheckStopReadingFn checkStopReadingFn); | ||
|
|
||
| abstract Builder<K, V> setBadRecordErrorHandler( | ||
| @Nullable ErrorHandler<BadRecord, ?> badRecordErrorHandler); |
Collaborator
There was a problem hiding this comment.
When will we ever put null in this parameter?
Contributor
Author
There was a problem hiding this comment.
We will never set it to be null. This is @nullable to indicate that the property may not be set, and that is valid
shunping
reviewed
Dec 6, 2023
| } catch (SerializationException e) { | ||
| // This exception should only occur during the key and value deserialization when | ||
| // creating the Kafka Record | ||
| badRecordRouter.route( |
Collaborator
There was a problem hiding this comment.
QQ: do we have timestamp in DLQ?
shunping
reviewed
Dec 6, 2023
Collaborator
shunping
left a comment
There was a problem hiding this comment.
Just a few comments and questions. Overall LGTM.
Contributor
Author
|
GCP IO Direct test is a flake, whitespace failure is due to files this PR doesn't touch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add .withBadRecordHandler() functionality to both KafkaIO.read() and .write()
This also adds usage to the Kafka streaming example
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.