Spark Runner : Fix invalid translateImpulse#34272
Conversation
|
Run Java PreCommit |
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
assign set of reviewers |
|
Assigning reviewers. If you would like to opt out of this review, comment R: @Abacn for label build. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Abacn
left a comment
There was a problem hiding this comment.
Thank you for quick fix!
btw, would you mind add a statement in https://github.com/apache/beam/blob/master/CHANGES.md#2640---unreleased to note the Spark runner improvement made here and in #34080 ?
|
Java Tests failed at uploading logs, infra flakiness, not related to the change |
Thank you for pointing that out! I completely forgot to update the CHANGES.md with the changes. I'll make sure to add the note regarding the Spark runner improvement made here and in #34080. Let me update it and create a pull request shortly. |
Please add a meaningful description for your change here
fixes #20426
This PR addresses an issue in the
translateImpulsemethod of theSparkStreamingPortablePipelineTranslator, which was introduced in the changes made under #34080.The root cause of the test failures was the use of
ConstantInputDStreaminstead ofSingleEmitInputDStream. This incorrect implementation led to the impulse being emitted multiple times, causing unexpected behavior.By replacing
ConstantInputDStreamwithSingleEmitInputDStream, this issue has been resolved.related PR : #34253
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.