feat: add schema aware stream writer by arturowczarek · Pull Request #2048 · googleapis/java-bigquerystorage

arturowczarek · 2023-03-20T10:12:00Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #2039 ☕️

If you write sample code, please follow the samples format.

arturowczarek · 2023-03-20T10:12:31Z

~~This is a work in progress~~. Please let me know what you think.

Neenu1995 · 2023-03-20T14:39:47Z

@yirutang PTAL

arturowczarek · 2023-03-20T16:00:30Z

+ * order of minutes).
+ */
+public class SchemaAwareStreamWriter<T> implements AutoCloseable {
+  private static String streamPatternString =


Fields like this should be final and CAPITALIZED, but I leave it this way. Otherwise I'd have to change too much.

arturowczarek · 2023-03-20T16:01:00Z

+ * table schema is updated, users will be able to ingest data on the new schema after some time (in
+ * order of minutes).
+ */
+public class SchemaAwareStreamWriter<T> implements AutoCloseable {


This class is intended to be a generic version of JsonStreamWriter

arturowczarek · 2023-03-20T16:01:46Z

+  private Descriptor descriptor;
+  private TableSchema tableSchema;
+  private boolean ignoreUnknownFields = false;
+  private boolean reconnectAfter10M = false;


It doesn't seem to be necessary. Can we remove it?

Yes, please.

arturowczarek · 2023-03-20T16:01:59Z

+    }
+
+    /**
+     * @Deprecated Setter for a reconnectAfter10M, temporaily workaround for omg/48020. Fix for the


Where do I find it? Can we remove it?

Please remove this.

arturowczarek · 2023-03-29T13:41:42Z

@yirutang @Neenu1995 Is there a chance for any feedback to this pull request and/or the created issues?

yirutang

Thanks for the addition!

yirutang · 2023-04-03T16:59:50Z

+  private Descriptor descriptor;
+  private TableSchema tableSchema;
+  private boolean ignoreUnknownFields = false;
+  private boolean reconnectAfter10M = false;


Yes, please.

yirutang · 2023-04-03T17:06:16Z

+    }
+
+    /**
+     * @Deprecated Setter for a reconnectAfter10M, temporaily workaround for omg/48020. Fix for the


Please remove this.

generated-files-bot · 2023-04-03T21:05:09Z

Warning: This pull request is touching the following templated files:

google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/package-info.java

arturowczarek · 2023-04-04T14:22:52Z

I think I've changed everything. The tests for JsonStreamWriter all pass. You can run the CI tests and do the review.

arturowczarek · 2023-04-05T10:40:32Z

Thank you @yirutang for running the tests again.
I've fixed the formatting. Sorry about that. I didn't notice kokoro requires additional environmental variable JOB_TYPE to know what to do.
I'm not sure what to do with clirr errors. It notices that I've introduced some changes to public interfaces: https://github.com/googleapis/java-bigquerystorage/actions/runs/4606032456/jobs/8148652677#step:5:3513
This includes renaming one of the class, removal of reconnectAfter10M, etc. I'm not sure what is the workflow and how do deal with it properly. Can you instruct me what to do? I've noticed @Neenu1995 commit e67e913, where she leaves the old, wrongly named class but leaves in the state that doesn't break the contract. When and how is it going to be removed? Should I also leave the classes and methods and mark them somehow that they are going to be removed?
I've committed 259391b to solve clirr errors. Please have a look if it is ok.
Now the tests on my laptop pass.

arturowczarek · 2023-04-06T08:31:12Z

@Neenu1995 Why didn't you finally use @Deprecated annotation in #2037?

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

arturowczarek · 2023-04-12T11:50:41Z

How does releasing of this library work? When does the releasing bot activate and release a new version?

arturowczarek requested a review from a team March 20, 2023 10:12

product-auto-label Bot added size: xl Pull request size is extra large. api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. labels Mar 20, 2023

arturowczarek force-pushed the generic_stream_writer branch 4 times, most recently from b2c069c to 2601bd4 Compare March 20, 2023 12:24

arturowczarek force-pushed the generic_stream_writer branch from 2601bd4 to acac9c0 Compare March 20, 2023 15:26

arturowczarek commented Mar 20, 2023

View reviewed changes

shollyman requested a review from yirutang March 30, 2023 15:48

yirutang approved these changes Apr 3, 2023

View reviewed changes

arturowczarek force-pushed the generic_stream_writer branch from acac9c0 to da908d1 Compare April 3, 2023 21:05

arturowczarek force-pushed the generic_stream_writer branch 9 times, most recently from ede37e8 to 2258a2a Compare April 4, 2023 09:12

feat: add schema aware stream writer

c42e1cb

arturowczarek force-pushed the generic_stream_writer branch from 2258a2a to c42e1cb Compare April 5, 2023 10:32

[squash this commit] Fix clirr errors

259391b

yirutang added the owlbot:run Add this label to trigger the Owlbot post processor. label Apr 7, 2023

gcf-owl-bot Bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Apr 7, 2023

yirutang added the kokoro:run Add this label to force Kokoro to re-run the tests. label Apr 7, 2023

yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Apr 7, 2023

🦉 Updates from OwlBot post-processor

d63008e

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

gcf-owl-bot Bot requested a review from a team April 7, 2023 00:38

yirutang added the kokoro:run Add this label to force Kokoro to re-run the tests. label Apr 7, 2023

yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Apr 7, 2023

yirutang merged commit ad136b9 into googleapis:main Apr 8, 2023

release-please Bot mentioned this pull request Apr 8, 2023

chore(main): release 2.35.0 #2074

Merged

bastewart mentioned this pull request Apr 21, 2023

JsonStreamWriter.Builder#setEnableConnectionPool does not obey argument passed #2083

Closed

Conversation

arturowczarek commented Mar 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arturowczarek commented Mar 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Neenu1995 commented Mar 20, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arturowczarek commented Mar 29, 2023

Uh oh!

yirutang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

generated-files-bot Bot commented Apr 3, 2023

Uh oh!

arturowczarek commented Apr 4, 2023

Uh oh!

arturowczarek commented Apr 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arturowczarek commented Apr 6, 2023

Uh oh!

arturowczarek commented Apr 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

arturowczarek commented Mar 20, 2023 •

edited

Loading

arturowczarek commented Mar 20, 2023 •

edited

Loading

arturowczarek commented Apr 5, 2023 •

edited

Loading