Skip to content

fix(bigquery): Implement some mitigations for Bigquery data and metadata delays#6381

Merged
hermanschaaf merged 6 commits intomainfrom
bigquery-migrations
Jan 6, 2023
Merged

fix(bigquery): Implement some mitigations for Bigquery data and metadata delays#6381
hermanschaaf merged 6 commits intomainfrom
bigquery-migrations

Conversation

@hermanschaaf
Copy link
Copy Markdown
Contributor

These changes make our tests pass more reliably, and is intended to help useres who are using the BigQuery plugin for the first time or migrating tables.

Changes:

  • Check schema metadata 3 times before accepting it as OK after a migration.
  • Retry writes if the returned error is that the table was not found. This will be done until the write timeout is reached (5 minutes) or some other error is returned.

@hermanschaaf hermanschaaf requested review from a team and candiduslynx and removed request for a team January 5, 2023 15:47
@hermanschaaf hermanschaaf changed the title fix: Implement some mitigations for Bigquery data and metadata delays fix(bigquery): Implement some mitigations for Bigquery data and metadata delays Jan 5, 2023
@cq-bot cq-bot added the bigquery label Jan 5, 2023
Copy link
Copy Markdown
Contributor

@yevgenypats yevgenypats left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🪄

@vercel
Copy link
Copy Markdown

vercel bot commented Jan 6, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated
cloudquery-web 🔄 Building (Inspect) Jan 6, 2023 at 1:04PM (UTC)

@hermanschaaf hermanschaaf merged commit ec1d132 into main Jan 6, 2023
@hermanschaaf hermanschaaf deleted the bigquery-migrations branch January 6, 2023 13:09
kodiakhq bot pushed a commit that referenced this pull request Jan 9, 2023
🤖 I have created a release *beep* *boop*
---


## [2.0.3](plugins-destination-bigquery-v2.0.2...plugins-destination-bigquery-v2.0.3) (2023-01-09)


### Bug Fixes

* **bigquery:** Implement some mitigations for Bigquery data and metadata delays ([#6381](#6381)) ([ec1d132](ec1d132))
* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.22.0 ([#6516](#6516)) ([b7e4e73](b7e4e73))
* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.23.0 ([#6522](#6522)) ([ce24f1d](ce24f1d))
* **deps:** Update plugin-sdk to v1.21.0 for destinations ([#6419](#6419)) ([f3b989f](f3b989f))
* Set batch size bytes limit in BigQuery ([#6439](#6439)) ([2f73cab](2f73cab))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants