Port all changes from the Pub/Sub Lite repo back to beam #15418

dpcollins-google · 2021-08-30T14:19:57Z

Beam will be the canonical source for this IO in the future per offline discussion.

Also add the other direction of helper to CloudPubsubTransforms and add an integration test.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

dpcollins-google · 2021-08-30T14:21:53Z

R: @aaltay

Ahmet, would you be willing to review this PR?

codecov · 2021-08-30T14:40:19Z

Codecov Report

Merging #15418 (b9a7339) into master (a89b12a) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #15418      +/-   ##
==========================================
- Coverage   83.76%   83.75%   -0.01%     
==========================================
  Files         443      443              
  Lines       60077    60101      +24     
==========================================
+ Hits        50325    50340      +15     
- Misses       9752     9761       +9

Impacted Files	Coverage Δ
...ks/python/apache_beam/runners/worker/data_plane.py	`87.70% <0.00%> (-1.68%)`	⬇️
sdks/python/apache_beam/io/source_test_utils.py	`88.47% <0.00%> (-1.39%)`	⬇️
...pache_beam/runners/interactive/interactive_beam.py	`75.53% <0.00%> (-1.07%)`	⬇️
...he_beam/runners/interactive/sql/beam_sql_magics.py	`54.70% <0.00%> (-0.96%)`	⬇️
...runners/interactive/display/pcoll_visualization.py	`85.26% <0.00%> (-0.53%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.51% <0.00%> (-0.25%)`	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	`88.80% <0.00%> (-0.16%)`	⬇️
sdks/python/apache_beam/pipeline.py	`91.66% <0.00%> (ø)`
sdks/python/apache_beam/io/gcp/bigquery.py	`75.53% <0.00%> (ø)`
sdks/python/apache_beam/transforms/util.py	`95.81% <0.00%> (ø)`
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a89b12a...b9a7339. Read the comment docs.

dpcollins-google · 2021-08-30T17:09:18Z

Build failures are unrelated.

aaltay · 2021-08-30T18:06:36Z

I do not think I am the best person to review this.

@chamikaramj @pabloem - Could one of you review this or suggest a reviewer?

chamikaramj · 2021-08-30T18:15:26Z

R: @pabloem

I'll pass to Pablo. @dpcollins-google pls add a Jira with the context.

dpcollins-google · 2021-08-31T20:01:29Z

Run Java PreCommit

dpcollins-google · 2021-08-31T20:01:40Z

Run SQL PreCommit

dpcollins-google · 2021-09-01T18:40:13Z

Run Java PreCommit

chamikaramj · 2021-09-07T17:52:33Z

Please add a Jira.

pabloem · 2021-09-08T18:36:06Z

Run Java PostCommit

...oogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PublisherCache.java

...o/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/ReadWriteIT.java

pabloem

Just added a couple comments. Thanks

pabloem · 2021-09-08T19:27:59Z

Run Java PostCommit

pabloem · 2021-09-08T19:28:26Z

Run Java PostCommit

pabloem · 2021-09-08T20:13:28Z

Run Java PostCommit

dpcollins-google · 2021-09-08T20:16:06Z

Run Java PostCommit

dpcollins-google · 2021-09-08T22:44:49Z

Run Java PostCommit

pabloem · 2021-09-08T23:04:58Z

note that the new test is failing:
https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/766/
https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/767/#showFailuresLink
https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/767/testReport/junit/org.apache.beam.sdk.io.gcp.pubsublite/ReadWriteIT/testReadWrite/

Beam will be the canonical source for this IO in the future per offline discussion. Also add the other direction of helper to CloudPubsubTransforms and add an integration test.

dpcollins-google · 2021-09-09T21:12:50Z

Run Java PostCommit

pabloem · 2021-09-09T22:06:20Z

Run Java PreCommit

pabloem · 2021-09-09T22:59:15Z

Run Java PreCommit

dpcollins-google · 2021-09-10T14:15:28Z

Run Java PostCommit

pabloem · 2021-09-10T17:31:36Z

Run Java PreCommit

pabloem · 2021-09-10T17:31:44Z

Run Java_Examples_Dataflow PreCommit

ibzib · 2021-09-14T19:44:16Z

@dpcollins-google The test pubsublite.SubscriptionPartitionLoaderTest.addedResults has been flaky for the past few days. Can you see if it is related to this change? https://issues.apache.org/jira/browse/BEAM-12882

* Port all changes from the Pub/Sub Lite repo back to beam. Beam will be the canonical source for this IO in the future per offline discussion. Also add the other direction of helper to CloudPubsubTransforms and add an integration test. * remove fixed TODO * Fixes to ReadWriteIT to work around Create or DirectRunner bug. * fix racy test (cherry picked from commit 8a646aa)

dpcollins-google mentioned this pull request Aug 30, 2021

Change PubsubLiteIO to reference com.google.cloud.pubsublite.beam.PubsubLiteIO from pubsublite-beam-io package #15076

Closed

4 tasks

aaltay requested a review from pabloem August 30, 2021 18:18

pabloem reviewed Sep 8, 2021

View reviewed changes

...oogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PublisherCache.java Outdated Show resolved Hide resolved

pabloem reviewed Sep 8, 2021

View reviewed changes

...o/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/ReadWriteIT.java Outdated Show resolved Hide resolved

pabloem reviewed Sep 8, 2021

View reviewed changes

dpcollins-google added 3 commits September 8, 2021 19:13

Port all changes from the Pub/Sub Lite repo back to beam.

96f53e8

Beam will be the canonical source for this IO in the future per offline discussion. Also add the other direction of helper to CloudPubsubTransforms and add an integration test.

remove fixed TODO

981db90

Fixes to ReadWriteIT to work around Create or DirectRunner bug.

e4432dd

dpcollins-google force-pushed the psl-backport-fixes branch from 579fdd7 to e4432dd Compare September 9, 2021 21:12

fix racy test

b9a7339

pabloem merged commit 8a646aa into apache:master Sep 10, 2021

udim added a commit that referenced this pull request Sep 30, 2021

Revert "Cherrypick #15418 and #15515"

bb5ecd1

steveniemitz pushed a commit to twitter-forks/beam that referenced this pull request Oct 18, 2021

Revert "Cherrypick apache#15418 and apache#15515"

a965b1e

Port all changes from the Pub/Sub Lite repo back to beam #15418

Port all changes from the Pub/Sub Lite repo back to beam #15418

Uh oh!

Conversation

dpcollins-google commented Aug 30, 2021

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

dpcollins-google commented Aug 30, 2021

Uh oh!

codecov bot commented Aug 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dpcollins-google commented Aug 30, 2021

Uh oh!

aaltay commented Aug 30, 2021

Uh oh!

chamikaramj commented Aug 30, 2021

Uh oh!

dpcollins-google commented Aug 31, 2021

Uh oh!

dpcollins-google commented Aug 31, 2021

Uh oh!

dpcollins-google commented Sep 1, 2021

Uh oh!

chamikaramj commented Sep 7, 2021

Uh oh!

pabloem commented Sep 8, 2021

Uh oh!

Uh oh!

Uh oh!

pabloem left a comment

Choose a reason for hiding this comment

Uh oh!

pabloem commented Sep 8, 2021

Uh oh!

pabloem commented Sep 8, 2021

Uh oh!

pabloem commented Sep 8, 2021

Uh oh!

dpcollins-google commented Sep 8, 2021

Uh oh!

dpcollins-google commented Sep 8, 2021

Uh oh!

pabloem commented Sep 8, 2021

Uh oh!

dpcollins-google commented Sep 9, 2021

Uh oh!

pabloem commented Sep 9, 2021

Uh oh!

pabloem commented Sep 9, 2021

Uh oh!

dpcollins-google commented Sep 10, 2021

Uh oh!

pabloem commented Sep 10, 2021

Uh oh!

pabloem commented Sep 10, 2021

Uh oh!

ibzib commented Sep 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

`ValidatesRunner` compliance status (on master branch)

codecov bot commented Aug 30, 2021 •

edited

Loading