Skip to content
This repository was archived by the owner on Feb 24, 2026. It is now read-only.

feat: add schema update support in JsonStreamWriter#1447

Merged
stephaniewang526 merged 12 commits intogoogleapis:mainfrom
stephaniewang526:schema-update
Dec 29, 2021
Merged

feat: add schema update support in JsonStreamWriter#1447
stephaniewang526 merged 12 commits intogoogleapis:mainfrom
stephaniewang526:schema-update

Conversation

@stephaniewang526
Copy link
Copy Markdown
Contributor

@stephaniewang526 stephaniewang526 commented Dec 15, 2021

No description provided.

@stephaniewang526 stephaniewang526 requested review from a team and loferris December 15, 2021 20:49
@product-auto-label product-auto-label bot added the api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. label Dec 15, 2021
@stephaniewang526 stephaniewang526 requested review from yirutang and removed request for loferris December 15, 2021 20:49
@stephaniewang526 stephaniewang526 added the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 16, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 16, 2021
@stephaniewang526 stephaniewang526 requested a review from a team December 16, 2021 12:30
yirutang
yirutang reviewed Dec 16, 2021
View reviewed changes
yirutang
yirutang reviewed Dec 22, 2021
View reviewed changes
...e/src/test/java/com/google/cloud/bigquery/storage/v1/it/ITBigQueryWriteManualClientTest.java
try {
response2 = jsonStreamWriter.append(jsonArr2);
next.getAndIncrement();
} catch (IOException | DescriptorValidationException e) {
Copy link
Copy Markdown
Contributor

@yirutang yirutang Dec 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully with ignore unknown fields, we could keep writing ccc and ddd to the table and expect ddd eventually to show up in the table. This is good for now.

Copy link
Copy Markdown
Contributor Author

@stephaniewang526 stephaniewang526 Dec 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. Will keep this in mind.

stephaniewang526 and others added 4 commits December 23, 2021 11:24
@stephaniewang526
Merge branch 'main' into schema-update
f43ac20
@stephaniewang526
update getUpdatedSchema() in StreamWriter.java to use lock; update co…
d3bf37f 
…ncurrent test case
@stephaniewang526
Merge branch 'main' into schema-update
1802c98 
# Conflicts:
#	google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java
@stephaniewang526
simplify logic
ee1d976
@stephaniewang526 stephaniewang526 added automerge Merge the pull request once unit tests and other checks pass. owlbot:run Add this label to trigger the Owlbot post processor. labels Dec 24, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 24, 2021
@stephaniewang526 stephaniewang526 added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed automerge Merge the pull request once unit tests and other checks pass. labels Dec 24, 2021
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 24, 2021
@stephaniewang526 stephaniewang526 added the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 28, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 28, 2021
shollyman
shollyman approved these changes Dec 28, 2021
View reviewed changes
...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java
// Create a new underlying StreamWriter with the updated TableSchema and Descriptor
this.streamWriter =
streamWriterBuilder
.setWriterSchema(ProtoSchemaConverter.convert(this.descriptor))
Copy link
Copy Markdown
Contributor

@shollyman shollyman Dec 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need to propagate anything else to the new writer like stream id, existing trace id, etc or does it all come from properties stored on the JsonStreamWriter?

Copy link
Copy Markdown
Contributor Author

@stephaniewang526 stephaniewang526 Dec 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, those things are needed and are already being populated in the streamWriterBuilder.

@stephaniewang526 stephaniewang526 merged commit 973afcc into googleapis:main Dec 29, 2021
@stephaniewang526 stephaniewang526 mentioned this pull request Dec 29, 2021
Merged
4 tasks
gcf-merge-on-green bot pushed a commit that referenced this pull request Jan 4, 2022
@release-please
chore: release 2.8.0 (#1448)
2b756e0 
🤖 I have created a release \*beep\* \*boop\*
---
## [2.8.0](https://www.github.com/googleapis/java-bigquerystorage/compare/v2.7.0...v2.8.0) (2021-12-30)


### Features

* Add ignoreUnknownField support in JsonWriter ([#1455](https://www.github.com/googleapis/java-bigquerystorage/issues/1455)) ([4616adb](https://www.github.com/googleapis/java-bigquerystorage/commit/4616adb30d07d7be522b74837adae080eca103a6))
* Add reconnect support to v1 client lib. ([#1446](https://www.github.com/googleapis/java-bigquerystorage/issues/1446)) ([a5157fa](https://www.github.com/googleapis/java-bigquerystorage/commit/a5157faf37354ee48a0af930a68edf4823b5be6f))
* add schema update support in JsonStreamWriter ([#1447](https://www.github.com/googleapis/java-bigquerystorage/issues/1447)) ([973afcc](https://www.github.com/googleapis/java-bigquerystorage/commit/973afccc22507b8109b3920d3fc183692dcc66c2))
* support `append()` without offset in `StreamWriter` ([#1452](https://www.github.com/googleapis/java-bigquerystorage/issues/1452)) ([cb8b0ad](https://www.github.com/googleapis/java-bigquerystorage/commit/cb8b0ad25fbf07c3509b3d94244a999b9cb90e54))


### Bug Fixes

* allow all fields to be null values ([#1450](https://www.github.com/googleapis/java-bigquerystorage/issues/1450)) ([e47ac79](https://www.github.com/googleapis/java-bigquerystorage/commit/e47ac791765486f02f027d3c440645ef2a379ba9))
* finish BIGNUMERIC support ([#1449](https://www.github.com/googleapis/java-bigquerystorage/issues/1449)) ([d9d51cd](https://www.github.com/googleapis/java-bigquerystorage/commit/d9d51cd203733a972c80e312e9e279b50fb260fa))
* fix a NullPtr when user closes a writer without connection being ever established ([#1454](https://www.github.com/googleapis/java-bigquerystorage/issues/1454)) ([b774f5d](https://www.github.com/googleapis/java-bigquerystorage/commit/b774f5d0d00275278708d1299b4912b455be4bed))
* update storageError support due to server side enhancement ([#1456](https://www.github.com/googleapis/java-bigquerystorage/issues/1456)) ([6243ad5](https://www.github.com/googleapis/java-bigquerystorage/commit/6243ad5cba61d4dae7f4ceb60b09c625e7589215))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
@release-please release-please bot mentioned this pull request Jan 4, 2022
Closed
@bphenriques bphenriques mentioned this pull request Jan 6, 2022
Closed
@damccorm damccorm mentioned this pull request Jun 4, 2022
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@yayi-google yayi-google Awaiting requested review from yayi-google

2 more reviewers

@shollyman shollyman shollyman approved these changes

@yirutang yirutang yirutang approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@stephaniewang526 @shollyman @yirutang @yoshi-kokoro