feat: Create Stream writer v2 - starting with core logics by yayi-google · Pull Request #867 · googleapis/java-bigquerystorage

yayi-google · 2021-02-21T01:40:14Z

Unblocks Dataflow sink test which unblocks customer update.

Compared to existing StreamWriter, its locking mechanism is much simpler.

...querystorage/src/test/java/com/google/cloud/bigquery/storage/v1beta2/StreamWriterV2Test.java

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/StreamWriterV2.java

codecov · 2021-02-22T20:29:24Z

Codecov Report

Merging #867 (eabad67) into master (9c103df) will increase coverage by 0.60%.
The diff coverage is 95.71%.

@@             Coverage Diff              @@
##             master     #867      +/-   ##
============================================
+ Coverage     80.03%   80.64%   +0.60%     
- Complexity      979     1005      +26     
============================================
  Files            74       76       +2     
  Lines          5290     5430     +140     
  Branches        408      420      +12     
============================================
+ Hits           4234     4379     +145     
- Misses          872      878       +6     
+ Partials        184      173      -11

Impacted Files	Coverage Δ	Complexity Δ
...cloud/bigquery/storage/v1beta2/StreamWriterV2.java	`94.95% <94.95%> (ø)`	`23.00 <23.00> (?)`
...oud/bigquery/storage/v1beta2/StreamConnection.java	`100.00% <100.00%> (ø)`	`3.00 <3.00> (?)`
...ery/storage/v1beta1/BaseBigQueryStorageClient.java	`72.88% <0.00%> (+1.69%)`	`22.00% <0.00%> (ø%)`
...ud/bigquery/storage/v1/BaseBigQueryReadClient.java	`60.97% <0.00%> (+2.43%)`	`12.00% <0.00%> (ø%)`
...gquery/storage/v1beta2/BaseBigQueryReadClient.java	`60.97% <0.00%> (+2.43%)`	`12.00% <0.00%> (ø%)`
.../bigquery/storage/v1beta2/BigQueryWriteClient.java	`75.38% <0.00%> (+4.61%)`	`33.00% <0.00%> (ø%)`
...bigquery/storage/v1alpha2/BigQueryWriteClient.java	`80.51% <0.00%> (+6.49%)`	`38.00% <0.00%> (ø%)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9c103df...b98d6be. Read the comment docs.

...rystorage/src/test/java/com/google/cloud/bigquery/storage/v1beta2/FakeBigQueryWriteImpl.java

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/StreamWriterV2.java

...querystorage/src/test/java/com/google/cloud/bigquery/storage/v1beta2/StreamWriterV2Test.java

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/StreamWriterV2.java

bigang-li · 2021-02-23T03:21:18Z

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/StreamWriterV2.java

+    log.info("User closing stream: " + streamName);
+    this.lock.lock();
+    try {
+      this.userClosed = true;


adding a local var to remember the previous value, so if user closed twice, the second time will be no_op (don't call the appendThread.join() multiple times).

bigang-li · 2021-02-23T03:22:18Z

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1beta2/StreamWriterV2.java

+
+      if (localQueue.isEmpty()) {
+        continue;
+      }


maybe be save a few lines of code here? it's useless according to context.

…onfig to v1.3.2 (googleapis#867) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-shared-config](https://togithub.com/googleapis/java-shared-config) | `1.3.0` -> `1.3.2` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-shared-config/1.3.2/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-shared-config/1.3.2/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-shared-config/1.3.2/compatibility-slim/1.3.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-shared-config/1.3.2/confidence-slim/1.3.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>googleapis/java-shared-config</summary> ### [`v1.3.2`](https://togithub.com/googleapis/java-shared-config/blob/HEAD/CHANGELOG.md#132-httpsgithubcomgoogleapisjava-shared-configcomparev131v132-2022-03-28) [Compare Source](https://togithub.com/googleapis/java-shared-config/compare/v1.3.1...v1.3.2) ### [`v1.3.1`](https://togithub.com/googleapis/java-shared-config/blob/HEAD/CHANGELOG.md#131-httpsgithubcomgoogleapisjava-shared-configcomparev130v131-2022-03-25) [Compare Source](https://togithub.com/googleapis/java-shared-config/compare/v1.3.0...v1.3.1) </details> --- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-storage-nio).

yayi-google added 2 commits February 20, 2021 17:35

Add a new StreamWriterV2.

b365503

Compared to existing StreamWriter, its locking mechanism is much simpler.

Merge remote-tracking branch 'upstream/master' into stream-writer-v2

d45416d

yayi-google requested review from stephaniewang526 and yirutang February 21, 2021 01:40

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Feb 21, 2021

product-auto-label bot added the api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. label Feb 21, 2021

yayi-google marked this pull request as ready for review February 21, 2021 01:59

yayi-google requested review from a team February 21, 2021 01:59

bigang-li reviewed Feb 21, 2021

View reviewed changes

yayi-google added 2 commits February 22, 2021 12:04

Stop using Java8 features as we still need to support Java7

3312402

Do not hold lock while sending requests, and some minor refactoring.

b98d6be

yirutang reviewed Feb 22, 2021

View reviewed changes

...querystorage/src/test/java/com/google/cloud/bigquery/storage/v1beta2/StreamWriterV2Test.java Show resolved Hide resolved

yirutang reviewed Feb 23, 2021

View reviewed changes

yirutang self-requested a review February 23, 2021 00:44

yirutang approved these changes Feb 23, 2021

View reviewed changes

bigang-li approved these changes Feb 23, 2021

View reviewed changes

stephaniewang526 approved these changes Feb 23, 2021

View reviewed changes

stephaniewang526 merged commit 7c01f45 into googleapis:master Feb 23, 2021

yayi-google deleted the stream-writer-v2 branch February 25, 2021 01:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Create Stream writer v2 - starting with core logics#867

feat: Create Stream writer v2 - starting with core logics#867
stephaniewang526 merged 4 commits intogoogleapis:masterfrom
yayi-google:stream-writer-v2

yayi-google commented Feb 21, 2021 •

edited by stephaniewang526

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Feb 22, 2021 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bigang-li Feb 23, 2021

Uh oh!

bigang-li Feb 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

yayi-google commented Feb 21, 2021 • edited by stephaniewang526 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Feb 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bigang-li Feb 23, 2021

Choose a reason for hiding this comment

Uh oh!

bigang-li Feb 23, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yayi-google commented Feb 21, 2021 •

edited by stephaniewang526

Loading

codecov bot commented Feb 22, 2021 •

edited

Loading