Support custom avro DatumReader when reading from BigQuery by kkdoon · Pull Request #22718 · apache/beam

kkdoon · 2022-08-13T05:28:45Z

Similar to 11479 this PR adds functionality to directly deserialize Avro records to the target class, via the user specified DatumReader, in BigQueryIO.

R: @pabloem @chamikaramj @steveniemitz

This fixes #22717

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

github-actions · 2022-08-13T06:05:01Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @kennknowles for label java.
R: @chamikaramj for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

steveniemitz · 2022-08-16T15:18:12Z

can we plumb this down into BigQueryStorageAvroReader as well so it'll work with direct reads?

steveniemitz · 2022-08-16T15:19:49Z

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

can we make the reader schema optional, and use the writer schema if its not set? Thats how AvroIO works IIRC.

It seems like AvroIO derives it in case the class is of SpecificData subType. We could do the same but we would need to take the class type as input instead (don't think its cleaner though).

Also, readerSchema is needed by AvroSource in general, as it does validation based on that, if parserFn is not set,
as well as derives the AvroCoder based on the readerSchema. Since it gets the writer schema during runtime from the file metadata, we cannot assign reader schema as writer schema on the submitter side.

kkdoon · 2022-08-16T17:03:29Z

BigQueryStorageAvroReader

yeah, i was thinking of tackling that in another review. I could add in this one too to allow this for different read types.

steveniemitz · 2022-08-16T17:34:25Z

BigQueryStorageAvroReader

yeah, i was thinking of tackling that in another review. I could add in this one too to allow this for different read types.

that's cool, you can do it in another review, just curious.

github-actions · 2022-08-24T12:15:36Z

Reminder, please take a look at this pr: @kennknowles @chamikaramj

kennknowles · 2022-08-25T21:08:13Z

R: @chamikaramj

github-actions · 2022-08-25T21:27:20Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

This reverts commit 80da4eff8bf674f3f27eee5733a9ec714dba1746.

This reverts commit b2b4737.

codecov · 2022-09-29T08:12:33Z

Codecov Report

Merging #22718 (1342a7f) into master (b59df6c) will increase coverage by 9.95%.
The diff coverage is n/a.

❗ Current head 1342a7f differs from pull request most recent head ec1ce6d. Consider uploading reports for the commit ec1ce6d to get more accurate results

@@            Coverage Diff             @@
##           master   #22718      +/-   ##
==========================================
+ Coverage   73.44%   83.40%   +9.95%     
==========================================
  Files         718      480     -238     
  Lines       95680    66483   -29197     
==========================================
- Hits        70273    55447   -14826     
+ Misses      24096    11036   -13060     
+ Partials     1311        0    -1311

Flag	Coverage Δ
go	`?`
python	`83.40% <ø> (+0.18%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ython/apache_beam/io/gcp/experimental/spannerio.py	`82.15% <0.00%> (-4.79%)`	⬇️
.../python/apache_beam/testing/test_stream_service.py	`88.09% <0.00%> (-4.77%)`	⬇️
...ks/python/apache_beam/runners/worker/statecache.py	`89.69% <0.00%> (-2.18%)`	⬇️
...che_beam/runners/interactive/interactive_runner.py	`90.06% <0.00%> (-1.71%)`	⬇️
...python/apache_beam/runners/worker/worker_status.py	`75.33% <0.00%> (-1.34%)`	⬇️
...ache_beam/runners/interactive/pipeline_fragment.py	`98.33% <0.00%> (-0.86%)`	⬇️
sdks/python/apache_beam/io/localfilesystem.py	`90.97% <0.00%> (-0.76%)`	⬇️
...eam/runners/portability/fn_api_runner/execution.py	`92.44% <0.00%> (-0.65%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.17% <0.00%> (-0.38%)`	⬇️
sdks/python/apache_beam/ml/inference/base.py	`95.58% <0.00%> (-0.26%)`	⬇️
... and 251 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

steveniemitz · 2022-09-29T16:12:09Z

Run Java PreCommit

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

steveniemitz · 2022-10-03T13:27:28Z

Run Java PreCommit

steveniemitz · 2022-10-03T13:27:46Z

Run Java_Examples_Dataflow PreCommit

steveniemitz · 2022-10-03T13:27:59Z

Run Java_Examples_Dataflow_Java11 PreCommit

steveniemitz · 2022-10-03T13:28:11Z

Run Java_Examples_Dataflow_Java17 PreCommit

steveniemitz · 2022-10-03T14:48:14Z

Run Java PreCommit

steveniemitz · 2022-10-05T20:49:19Z

Run Java PreCommit

kkdoon · 2022-10-06T15:09:52Z

I ran and profiled a few dataflow jobs with and without my changes to test for performance regression. Overall the profiling data looks good for these new changes.

ahmedabu98 · 2022-10-11T18:28:07Z

I believe this PR broke Java Dataflow PostCommits: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV1/2173/testReport/. See #23541

github-actions bot added gcp io java labels Aug 13, 2022

github-actions bot added the Next Action: Reviewers label Aug 13, 2022

steveniemitz requested review from chamikaramj and steveniemitz August 16, 2022 15:07

steveniemitz reviewed Aug 16, 2022

View reviewed changes

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java Outdated Show resolved Hide resolved

steveniemitz reviewed Aug 16, 2022

View reviewed changes

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java Outdated Show resolved Hide resolved

steveniemitz reviewed Aug 16, 2022

View reviewed changes

github-actions bot added the slow-review label Aug 24, 2022

github-actions bot removed the slow-review label Aug 25, 2022

github-actions bot added the build label Sep 1, 2022

github-actions bot added build core dataflow docker examples extensions fn-execution go hadoop hcatalog and removed build labels Sep 29, 2022

Kanishk Karanawat added 4 commits September 29, 2022 00:53

rebeased 2

8743c5d

Revert "rebeased 2"

b656a42

This reverts commit 80da4eff8bf674f3f27eee5733a9ec714dba1746.

Revert "added readWithDatumReader api for bqio"

04d1b8e

This reverts commit b2b4737.

update test

246a3aa

kkdoon reopened this Sep 29, 2022

steveniemitz reviewed Sep 29, 2022

View reviewed changes

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java Outdated Show resolved Hide resolved

steveniemitz reviewed Sep 29, 2022

View reviewed changes

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java Outdated Show resolved Hide resolved

Kanishk Karanawat added 2 commits September 29, 2022 23:08

remove readerSchema related changes

054da5c

subclass GenericDatumTransformer as DatumReader instead

9a16eb8

steveniemitz reviewed Sep 30, 2022

View reviewed changes

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java Show resolved Hide resolved

steveniemitz reviewed Sep 30, 2022

View reviewed changes

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java Outdated Show resolved Hide resolved

Kanishk Karanawat added 4 commits September 30, 2022 13:19

remove readerSchema from public API and fix setSchema implementation

d065e43

nit: fix todo comment

d49ce57

refactor api name to avoid kotlin overload issue

5377971

spotless

e16b883

Kanishk Karanawat added 2 commits October 4, 2022 23:51

handle tableSchema serialization issue

30a2da1

re-organize fields in unit test

ec1ce6d

kkdoon mentioned this pull request Oct 12, 2022

Use writer schema only for BigQueryIO Read API #23594

Merged

4 tasks

piter75 mentioned this pull request Apr 18, 2023

[Bug]: BigQuerySourceBase does not propagate a Coder to AvroSource #26329

Open

15 tasks

Abacn mentioned this pull request Sep 13, 2023

Remove TableSchema to JSON conversion. #28274

Merged

1 task

Conversation

kkdoon commented Aug 13, 2022 • edited by steveniemitz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

github-actions bot commented Aug 13, 2022

Uh oh!

Uh oh!

Uh oh!

steveniemitz commented Aug 16, 2022

Uh oh!

steveniemitz Aug 16, 2022

Choose a reason for hiding this comment

Uh oh!

kkdoon Aug 16, 2022

Choose a reason for hiding this comment

Uh oh!

kkdoon commented Aug 16, 2022

Uh oh!

steveniemitz commented Aug 16, 2022

Uh oh!

github-actions bot commented Aug 24, 2022

Uh oh!

kennknowles commented Aug 25, 2022

Uh oh!

github-actions bot commented Aug 25, 2022

Uh oh!

codecov bot commented Sep 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

steveniemitz commented Sep 29, 2022

Uh oh!

Uh oh!

Uh oh!

steveniemitz commented Oct 3, 2022

Uh oh!

steveniemitz commented Oct 3, 2022

Uh oh!

steveniemitz commented Oct 3, 2022

Uh oh!

steveniemitz commented Oct 3, 2022

Uh oh!

steveniemitz commented Oct 3, 2022

Uh oh!

steveniemitz commented Oct 5, 2022

Uh oh!

kkdoon commented Oct 6, 2022

Uh oh!

ahmedabu98 commented Oct 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kkdoon commented Aug 13, 2022 •

edited by steveniemitz

Loading

codecov bot commented Sep 29, 2022 •

edited

Loading