[BEAM-6611] BigQuery file loads in Streaming for Python SDK by ttanay · Pull Request #8871 · apache/beam

ttanay · 2019-06-16T19:35:03Z

Added support for writing to BigQuery via file loads for the Python SDK.
Main PR towards my GSoC project.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang	Apex	Dataflow	Gearpump	Samza
Go	---	---	---	---
Java
Python	---		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

ttanay · 2019-06-17T12:53:55Z

Run Python PostCommit

ttanay · 2019-06-23T05:03:35Z

Run Python PostCommit

ttanay · 2019-06-24T20:00:27Z

Run Portable_Python PreCommit

ttanay · 2019-06-24T22:25:22Z

Run Python PostCommit

ttanay · 2019-06-27T18:09:16Z

Run Python PostCommit

ttanay · 2019-06-27T18:14:08Z

Run Python PostCommit

ttanay · 2019-06-28T14:17:17Z

Run Python PostCommit

ttanay · 2019-07-02T09:50:07Z

Run Python PostCommit

ttanay · 2019-07-02T11:55:14Z

This depends on #8934.
This makes use of a BigQuery matcher that polls BQ for results, that was introduced in #8934.

ttanay · 2019-07-09T16:35:24Z

Run Python PostCommit

ttanay · 2019-07-11T13:58:47Z

Run Python PreCommit

ttanay · 2019-07-11T18:34:54Z

Run Python PreCommit

ttanay · 2019-07-11T20:16:02Z

Run Python PreCommit

ttanay · 2019-07-12T20:33:28Z

Run Python PostCommit

ttanay · 2019-07-13T17:58:53Z

Run Python PreCommit

ttanay · 2019-07-13T18:41:10Z

Run Python PreCommit

ttanay · 2019-07-13T18:56:11Z

Run Python PostCommit

ttanay · 2019-07-20T19:13:26Z

Run Python 3.5 PostCommit

ttanay · 2019-07-20T19:13:33Z

Run Python 3.6 PostCommit

ttanay · 2019-07-20T19:13:42Z

Run Python 3.7 PostCommit

ttanay · 2019-07-20T21:01:37Z

Run Python 2 PostCommit

When running tests in parallel with nose, it has a caveat - it may find and run more tests than were specified. https://nose.readthedocs.io/en/latest/doc_tests/test_multiprocess/multiprocess.html#other-differences-in-test-running This change/hack prevents this from happening.

ttanay · 2019-07-21T17:22:39Z

Run Python 2 PostCommit

ttanay · 2019-07-21T20:51:44Z

Run Python 2 PostCommit

ttanay · 2019-07-21T22:01:45Z

The failure in Python PreCommit is die to task - :sdks:python:docs.
The failure in Python 2 PostCommit is due to task - :sdks:python:crossLanguagePortableWordCount.
All tests in task :sdks:python:directRunnerIT pass ☑️

ttanay · 2019-07-22T14:46:31Z

Run Python 2 PostCommit

Add extra triggers for streaming to cater to streams with large no of files. Use same method to write files in batch and streaming

ttanay · 2019-07-22T14:49:44Z

Run Python 2 PostCommit

ttanay · 2019-07-22T14:49:55Z

Run Python_PVR_Flink PreCommit

ttanay · 2019-07-22T16:26:49Z

Run Python 2 PostCommit

ttanay · 2019-07-22T17:30:43Z

The same test - apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python is failing on the master as well. The streaming tests in :sdks:python:directRunnerIT aren't being run due to it.

ttanay · 2019-07-23T08:32:20Z

Run Python 2 PostCommit

sdks/python/apache_beam/io/gcp/bigquery.py

ttanay · 2019-07-25T21:06:23Z

Run Python 2 PostCommit

pabloem · 2019-07-25T21:59:38Z

This looks good to me Tanay. The test looks good too. Let's add any pending changes, and ensure the tests pass - and we can merge.

ttanay · 2019-07-26T13:08:13Z

I've made the remaining changes:

Limitations comment
Change in load job names

All tests pass ✔️
We can merge :)

ttanay · 2019-07-27T11:32:12Z

Thank You @pabloem!

ttanay force-pushed the beam-6611 branch 2 times, most recently from cc1e01d to 2f510bf Compare June 17, 2019 11:05

ttanay force-pushed the beam-6611 branch 4 times, most recently from f517951 to bb16f24 Compare June 22, 2019 21:53

ttanay force-pushed the beam-6611 branch from bb16f24 to 285c740 Compare June 24, 2019 19:51

ttanay force-pushed the beam-6611 branch from 285c740 to 862519c Compare June 24, 2019 21:44

ttanay mentioned this pull request Jul 9, 2019

[Beam 6611] Sketch for Integration Tests(failing) for BQ file loads in streaming #8763

Closed

3 tasks

ttanay force-pushed the beam-6611 branch from eca94d0 to 83e39c4 Compare July 11, 2019 13:47

ttanay force-pushed the beam-6611 branch from 83e39c4 to 2ad3bdf Compare July 11, 2019 18:41

ttanay force-pushed the beam-6611 branch from 2ad3bdf to 3a4d762 Compare July 12, 2019 13:22

ttanay force-pushed the beam-6611 branch from 3a4d762 to a11b1aa Compare July 13, 2019 14:06

ttanay added 2 commits July 21, 2019 22:20

[BEAM-6611] BigQuery file loads in streaming for Python SDK

0a43d3a

ttanay force-pushed the beam-6611 branch from 5ed2ed3 to 6a25f0b Compare July 21, 2019 17:20

ttanay force-pushed the beam-6611 branch from 6a25f0b to d300560 Compare July 22, 2019 14:45

ttanay added 4 commits July 22, 2019 20:18

fixup! [BEAM-6611] BigQuery file loads in streaming for Python SDK

34b6e31

Add extra triggers for streaming to cater to streams with large no of files. Use same method to write files in batch and streaming

Add triggering_frequency to docstring

cbfdaeb

Raise error when using triggering_frequency with streaming inserts

7798d5f

Add logging to BigqueryFullResultStreamingMatcher

78f0c09

ttanay force-pushed the beam-6611 branch from d300560 to 78f0c09 Compare July 22, 2019 14:49

ttanay commented Jul 23, 2019

View reviewed changes

sdks/python/apache_beam/io/gcp/bigquery.py Show resolved Hide resolved

ttanay added 2 commits July 26, 2019 00:24

Load job name contains int timestamp

3477b44

Add [BEAM-7822] to limitations

670a3bd

pabloem merged commit 6cf3b11 into apache:master Jul 27, 2019

Conversation

ttanay commented Jun 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

Uh oh!

ttanay commented Jun 17, 2019

Uh oh!

ttanay commented Jun 23, 2019

Uh oh!

ttanay commented Jun 24, 2019

Uh oh!

ttanay commented Jun 24, 2019

Uh oh!

ttanay commented Jun 27, 2019

Uh oh!

ttanay commented Jun 27, 2019

Uh oh!

ttanay commented Jun 28, 2019

Uh oh!

ttanay commented Jul 2, 2019

Uh oh!

ttanay commented Jul 2, 2019

Uh oh!

ttanay commented Jul 9, 2019

Uh oh!

ttanay commented Jul 11, 2019

Uh oh!

ttanay commented Jul 11, 2019

Uh oh!

ttanay commented Jul 11, 2019

Uh oh!

ttanay commented Jul 12, 2019

Uh oh!

ttanay commented Jul 13, 2019

Uh oh!

ttanay commented Jul 13, 2019

Uh oh!

ttanay commented Jul 13, 2019

Uh oh!

ttanay commented Jul 20, 2019

Uh oh!

ttanay commented Jul 20, 2019

Uh oh!

ttanay commented Jul 20, 2019

Uh oh!

ttanay commented Jul 20, 2019

Uh oh!

ttanay commented Jul 21, 2019

Uh oh!

ttanay commented Jul 21, 2019

Uh oh!

ttanay commented Jul 21, 2019

Uh oh!

ttanay commented Jul 22, 2019

Uh oh!

ttanay commented Jul 22, 2019

Uh oh!

ttanay commented Jul 22, 2019

Uh oh!

ttanay commented Jul 22, 2019

Uh oh!

ttanay commented Jul 22, 2019

Uh oh!

ttanay commented Jul 23, 2019

Uh oh!

Uh oh!

ttanay commented Jul 25, 2019

Uh oh!

pabloem commented Jul 25, 2019

Uh oh!

ttanay commented Jul 26, 2019

Uh oh!

ttanay commented Jul 27, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

ttanay commented Jun 16, 2019 •

edited

Loading