Skip to content

[BEAM-9795] Support custom avro DatumWriters when writing to BigQuery#11479

Merged
pabloem merged 1 commit intoapache:masterfrom
tc-dc:bq-avro-data
Apr 30, 2020
Merged

[BEAM-9795] Support custom avro DatumWriters when writing to BigQuery#11479
pabloem merged 1 commit intoapache:masterfrom
tc-dc:bq-avro-data

Conversation

@steveniemitz
Copy link
Copy Markdown
Contributor

@steveniemitz steveniemitz commented Apr 21, 2020

There are cases where we'd like to further control the user record -> avro process. For example, I might not want to write generic records, but instead use my own DatumWriter instance to do the avro writing.

This PR simply allows further customizing the AvroRowWriter that is created during pipeline construction, allowing the user to hook into the DatumWriter construction.

R: @pabloem @chamikaramj

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status
Build Status		 Build Status
Build Status
Build Status
Build Status		 Build Status Build Status Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status		 --- Build Status
Build Status
Build Status		 Build Status
Build Status		 --- --- Build Status
XLang --- --- --- Build Status --- --- Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status		 Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@steveniemitz
Copy link
Copy Markdown
Contributor Author

steveniemitz commented Apr 21, 2020

hm, looks like the precommit is failing with:

13:01:36 FAILURE: Build failed with an exception.
13:01:36 
13:01:36 * What went wrong:
13:01:36 Execution failed for task ':sdks:java:container:generateThirdPartyLicenses'.
13:01:36 > Process 'command './sdks/java/container/license_scripts/license_script.sh'' finished with non-zero exit value 1

which hopefully my change didn't break ;)

@chamikaramj chamikaramj requested a review from pabloem April 22, 2020 19:30
@pabloem
Copy link
Copy Markdown
Member

pabloem commented Apr 22, 2020

Hi Steve! thanks for the feature. I'll take a look hopefully by friday

@steveniemitz
Copy link
Copy Markdown
Contributor Author

steveniemitz commented Apr 22, 2020

thanks!

@steveniemitz
Copy link
Copy Markdown
Contributor Author

steveniemitz commented Apr 30, 2020

bump?

@pabloem
Copy link
Copy Markdown
Member

pabloem commented Apr 30, 2020

ah yes sorry Steve. Thanks for the contribution. This looks good to me. I'll just rerun precomits to get a green run, and run postcommits as well.

@pabloem
Copy link
Copy Markdown
Member

pabloem commented Apr 30, 2020

Run Java PostCommit

@pabloem
Copy link
Copy Markdown
Member

pabloem commented Apr 30, 2020

Run Java PreCommit

@steveniemitz
Copy link
Copy Markdown
Contributor Author

steveniemitz commented Apr 30, 2020 •
edited
Loading

thanks! I'm going to have a similar PR for AvroIO coming up, do you know who's best to review that?

edit: hmm looks like this pre-commit broke again w/ the license error, do I need to rebase this?

@pabloem
Copy link
Copy Markdown
Member

pabloem commented Apr 30, 2020

since it's only that task that failed, we can merge anyway.

For AvroIO - I think @iemejia may be a good candidate.

@pabloem pabloem merged commit 144b96d into apache:master Apr 30, 2020
@pabloem
Copy link
Copy Markdown
Member

pabloem commented Apr 30, 2020

thanks @steveniemitz !

@steveniemitz steveniemitz deleted the bq-avro-data branch May 8, 2020 15:05
@steveniemitz steveniemitz mentioned this pull request May 8, 2020
Merged
4 tasks
@steveniemitz steveniemitz mentioned this pull request Jul 17, 2020
Merged
3 tasks
@kkdoon kkdoon mentioned this pull request Aug 13, 2022
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pabloem pabloem Awaiting requested review from pabloem

Assignees

No one assigned

Labels

gcp io java

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@steveniemitz @pabloem