Change PubsubLiteIO to reference com.google.cloud.pubsublite.beam.PubsubLiteIO from pubsublite-beam-io package #15076

dpcollins-google · 2021-06-24T19:08:42Z

Also remove all implementation of the I/O from the beam repo.

Also add an integration test of core read and write behavior.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Samza	Twister2
Go	---	---	---
Java
Python	---	---	---
XLang		---	---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

…subLiteIO from pubsublite-beam-io package Also remove all implementation of the I/O from the beam repo.

codecov · 2021-06-24T23:07:27Z

Codecov Report

Merging #15076 (e1c1b8b) into master (a1950ef) will increase coverage by 0.00%.
The diff coverage is 72.72%.

@@           Coverage Diff            @@
##           master   #15076    +/-   ##
========================================
  Coverage   83.76%   83.76%            
========================================
  Files         439      439            
  Lines       59049    59176   +127     
========================================
+ Hits        49463    49571   +108     
- Misses       9586     9605    +19

Impacted Files	Coverage Δ
...dks/python/apache_beam/options/pipeline_options.py	`95.15% <ø> (ø)`
...apache_beam/runners/dataflow/internal/apiclient.py	`74.66% <50.00%> (-0.09%)`	⬇️
sdks/python/apache_beam/transforms/trigger.py	`89.27% <50.00%> (-0.02%)`	⬇️
sdks/python/apache_beam/transforms/core.py	`91.88% <85.71%> (-0.11%)`	⬇️
sdks/python/apache_beam/io/mongodbio.py	`92.88% <0.00%> (-2.76%)`	⬇️
sdks/python/apache_beam/pvalue.py	`91.35% <0.00%> (-1.67%)`	⬇️
...ython/apache_beam/io/gcp/bigquery_read_internal.py	`58.46% <0.00%> (-0.75%)`	⬇️
...eam/runners/interactive/caching/streaming_cache.py	`95.35% <0.00%> (-0.35%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.73% <0.00%> (-0.22%)`	⬇️
sdks/python/apache_beam/io/range_trackers.py	`94.14% <0.00%> (-0.20%)`	⬇️
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2088b66...e1c1b8b. Read the comment docs.

anguillanneuf · 2021-07-02T00:41:27Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

        proto_google_cloud_spanner_v1: "com.google.api.grpc:proto-google-cloud-spanner-v1", // google_cloud_platform_libraries_bom sets version
        proto_google_cloud_spanner_admin_database_v1: "com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1", // google_cloud_platform_libraries_bom sets version
        proto_google_common_protos                  : "com.google.api.grpc:proto-google-common-protos", // google_cloud_platform_libraries_bom sets version
+        pubsublite_beam_io                          : "com.google.cloud:pubsublite-beam-io:$google_cloud_pubsublite_version",


It'd be better to put it next to all the other google-cloud clients:

beam/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Line 535 in e1c1b8b

google_cloud_pubsublite : "com.google.cloud:google-cloud-pubsublite:$google_cloud_pubsublite_version",

I'll change the variable name to "google_cloud_pubsublite_beam_io" so it can both remain alphabetical and be sorted up there.

anguillanneuf

Thank you for moving things forward with this PR.

I only see one integration test - PubsubLiteReadIT. Are you planning to add a WriteIT as well?

dpcollins-google · 2021-07-02T22:17:30Z

Thank you for moving things forward with this PR.

I only see one integration test - PubsubLiteReadIT. Are you planning to add a WriteIT as well?

This IT is actually cloned from the pubsublite repo- it performs both a read and write with PubsubLiteIO. Going to rename to ReadWriteIT instead.

aaltay · 2021-07-16T03:44:37Z

I assume this is still pending the conclusion of the mailing list discussion?

dpcollins-google · 2021-08-30T14:20:40Z

Closed to prefer #15418

Change PubsubLiteIO to reference com.google.cloud.pubsublite.beam.Pub…

97da8c8

…subLiteIO from pubsublite-beam-io package Also remove all implementation of the I/O from the beam repo.

This was referenced Jun 24, 2021

Copy over Pub/Sub Lite changes from java-pubsublite repo. #14976

Closed

Extract PubsubMessage transforms in PubsubMessages and use them to add a compatability transform for Pub/Sub Lite. #14917

Closed

dpcollins-google added 3 commits June 24, 2021 16:17

add import

e572bff

fix lint errors

6dcf68a

fix sql

e1c1b8b

anguillanneuf reviewed Jul 2, 2021

View reviewed changes

anguillanneuf suggested changes Jul 2, 2021

View reviewed changes

dpcollins-google closed this Aug 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Change PubsubLiteIO to reference com.google.cloud.pubsublite.beam.PubsubLiteIO from pubsublite-beam-io package #15076

Change PubsubLiteIO to reference com.google.cloud.pubsublite.beam.PubsubLiteIO from pubsublite-beam-io package #15076

Uh oh!

dpcollins-google commented Jun 24, 2021

Uh oh!

codecov bot commented Jun 24, 2021 •

edited

Loading

Uh oh!

anguillanneuf Jul 2, 2021

Uh oh!

dpcollins-google Jul 2, 2021

Uh oh!

anguillanneuf left a comment •

edited

Loading

Uh oh!

dpcollins-google commented Jul 2, 2021

Uh oh!

aaltay commented Jul 16, 2021

Uh oh!

dpcollins-google commented Aug 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Change PubsubLiteIO to reference com.google.cloud.pubsublite.beam.PubsubLiteIO from pubsublite-beam-io package #15076

Change PubsubLiteIO to reference com.google.cloud.pubsublite.beam.PubsubLiteIO from pubsublite-beam-io package #15076

Uh oh!

Conversation

dpcollins-google commented Jun 24, 2021

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

codecov bot commented Jun 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

anguillanneuf Jul 2, 2021

Choose a reason for hiding this comment

Uh oh!

dpcollins-google Jul 2, 2021

Choose a reason for hiding this comment

Uh oh!

anguillanneuf left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dpcollins-google commented Jul 2, 2021

Uh oh!

aaltay commented Jul 16, 2021

Uh oh!

dpcollins-google commented Aug 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

`ValidatesRunner` compliance status (on master branch)

codecov bot commented Jun 24, 2021 •

edited

Loading

anguillanneuf left a comment •

edited

Loading