Skip to content
This repository was archived by the owner on Feb 24, 2026. It is now read-only.

feat: Add ignoreUnknownField support in JsonWriter#1455

Merged
gcf-merge-on-green[bot] merged 5 commits intogoogleapis:mainfrom
yirutang:ignoreUnknown
Dec 29, 2021
Merged

feat: Add ignoreUnknownField support in JsonWriter#1455
gcf-merge-on-green[bot] merged 5 commits intogoogleapis:mainfrom
yirutang:ignoreUnknown

Conversation

@yirutang
Copy link
Copy Markdown
Contributor

@yirutang yirutang commented Dec 28, 2021

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

@yirutang yirutang requested review from a team and tswast December 28, 2021 21:38
@product-auto-label product-auto-label bot added the api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. label Dec 28, 2021
@stephaniewang526 stephaniewang526 added the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 29, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 29, 2021
@stephaniewang526
Copy link
Copy Markdown
Contributor

stephaniewang526 commented Dec 29, 2021

We should revisit the test case for schema update while introducing this feat (related to this comment).

yirutang
yirutang commented Dec 29, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor Author

@yirutang yirutang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL, rebased.

@stephaniewang526 stephaniewang526 added the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 29, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 29, 2021
stephaniewang526
stephaniewang526 suggested changes Dec 29, 2021
View reviewed changes
...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java Outdated
*
* @return Builder
*/
public Builder ignoreUnknownFields() {
Copy link
Copy Markdown
Contributor

@stephaniewang526 stephaniewang526 Dec 29, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's more conventional and explicit(true vs. false) if we do setIgnoreUnknownFields(boolean ignoreUnknwonFields) {...} (an equivalent example in google-cloud-bigquery client).

Copy link
Copy Markdown
Contributor Author

@yirutang yirutang Dec 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

...d-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java

return convertJsonToProtoMessageImpl(protoSchema, null, json, "root", /*topLevel=*/ true);
return convertJsonToProtoMessageImpl(
protoSchema, null, json, "root", /*topLevel=*/ true, false);
Copy link
Copy Markdown
Contributor

@stephaniewang526 stephaniewang526 Dec 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline, topLevel is not used in convertJsonToProtoMessageImpl. Hence, we should remove it.

Copy link
Copy Markdown
Contributor Author

@yirutang yirutang Dec 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack

...d-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java
if (field == null && !ignoreUnknownFields) {
throw new IllegalArgumentException(
String.format("JSONObject has fields unknown to BigQuery: %s.", currentScope));
} else if (field == null) {
Copy link
Copy Markdown
Contributor

@stephaniewang526 stephaniewang526 Dec 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check seems redundant.

Copy link
Copy Markdown
Contributor Author

@yirutang yirutang Dec 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it is null, it means we cannot proceed on further processing. Otherwise, null ptr will be thrown if we continue.

Copy link
Copy Markdown
Contributor

@stephaniewang526 stephaniewang526 Dec 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay makes sense -- this allows us to skip the unknown field in the middle and proceed onto other fields.

stephaniewang526
stephaniewang526 approved these changes Dec 29, 2021
View reviewed changes
...d-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java
if (field == null && !ignoreUnknownFields) {
throw new IllegalArgumentException(
String.format("JSONObject has fields unknown to BigQuery: %s.", currentScope));
} else if (field == null) {
Copy link
Copy Markdown
Contributor

@stephaniewang526 stephaniewang526 Dec 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay makes sense -- this allows us to skip the unknown field in the middle and proceed onto other fields.

@stephaniewang526 stephaniewang526 added the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 29, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 29, 2021
@stephaniewang526 stephaniewang526 added automerge Merge the pull request once unit tests and other checks pass. owlbot:run Add this label to trigger the Owlbot post processor. labels Dec 29, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 29, 2021
@gcf-merge-on-green gcf-merge-on-green bot merged commit 4616adb into googleapis:main Dec 29, 2021
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Dec 29, 2021
gcf-merge-on-green bot pushed a commit that referenced this pull request Jan 4, 2022
@release-please
chore: release 2.8.0 (#1448)
2b756e0 
🤖 I have created a release \*beep\* \*boop\*
---
## [2.8.0](https://www.github.com/googleapis/java-bigquerystorage/compare/v2.7.0...v2.8.0) (2021-12-30)


### Features

* Add ignoreUnknownField support in JsonWriter ([#1455](https://www.github.com/googleapis/java-bigquerystorage/issues/1455)) ([4616adb](https://www.github.com/googleapis/java-bigquerystorage/commit/4616adb30d07d7be522b74837adae080eca103a6))
* Add reconnect support to v1 client lib. ([#1446](https://www.github.com/googleapis/java-bigquerystorage/issues/1446)) ([a5157fa](https://www.github.com/googleapis/java-bigquerystorage/commit/a5157faf37354ee48a0af930a68edf4823b5be6f))
* add schema update support in JsonStreamWriter ([#1447](https://www.github.com/googleapis/java-bigquerystorage/issues/1447)) ([973afcc](https://www.github.com/googleapis/java-bigquerystorage/commit/973afccc22507b8109b3920d3fc183692dcc66c2))
* support `append()` without offset in `StreamWriter` ([#1452](https://www.github.com/googleapis/java-bigquerystorage/issues/1452)) ([cb8b0ad](https://www.github.com/googleapis/java-bigquerystorage/commit/cb8b0ad25fbf07c3509b3d94244a999b9cb90e54))


### Bug Fixes

* allow all fields to be null values ([#1450](https://www.github.com/googleapis/java-bigquerystorage/issues/1450)) ([e47ac79](https://www.github.com/googleapis/java-bigquerystorage/commit/e47ac791765486f02f027d3c440645ef2a379ba9))
* finish BIGNUMERIC support ([#1449](https://www.github.com/googleapis/java-bigquerystorage/issues/1449)) ([d9d51cd](https://www.github.com/googleapis/java-bigquerystorage/commit/d9d51cd203733a972c80e312e9e279b50fb260fa))
* fix a NullPtr when user closes a writer without connection being ever established ([#1454](https://www.github.com/googleapis/java-bigquerystorage/issues/1454)) ([b774f5d](https://www.github.com/googleapis/java-bigquerystorage/commit/b774f5d0d00275278708d1299b4912b455be4bed))
* update storageError support due to server side enhancement ([#1456](https://www.github.com/googleapis/java-bigquerystorage/issues/1456)) ([6243ad5](https://www.github.com/googleapis/java-bigquerystorage/commit/6243ad5cba61d4dae7f4ceb60b09c625e7589215))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
@release-please release-please bot mentioned this pull request Jan 4, 2022
Closed
@stephaniewang526 stephaniewang526 mentioned this pull request Jan 9, 2022
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@tswast tswast Awaiting requested review from tswast

1 more reviewer

@stephaniewang526 stephaniewang526 stephaniewang526 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@yirutang @stephaniewang526