feat: update synth to generate v1beta2, v1 endpoints for bigquerystorage #10
Conversation
There's also some work to try to accomodate v1alpha2 in this PR, but there exists further oddities that must be tackled before we can fully add generation of the alpha client here. Also, note that this PR does not include manual client modifications such as streaming offset resumption that are present in the v1beta1 client. Intent is to address further changes like that in subsequent PRs.
tswast
left a comment
There was a problem hiding this comment.
Changes to synth.py LGTM.
Are you planning to handle the handwritten "reader" module and manual client wrappers in a subsequent PR?
| } | ||
| if version in ["v1beta2","v1"]: | ||
| clientinfo = { | ||
| "file": "big_query_read_client.py", |
There was a problem hiding this comment.
Am I understanding this correctly that
v1beta2andv1only have a read clientv1alpha2only has a write client- and
v1beta1only has a read client, but it's called a storage client?
There was a problem hiding this comment.
That's the gist of it, yes. I expect once v1alpha2 revs (e.g. to...v2beta1 or similar) we'll have both a read and write client in the same library/version.
| from google.cloud.bigquery_storage_v1beta1 import BigQueryStorageClient | ||
| from google.cloud.bigquery_storage_v1beta1 import enums | ||
| from google.cloud.bigquery_storage_v1beta1 import types | ||
| from google.cloud.bigquery_storage_v1 import BigQueryReadClient |
There was a problem hiding this comment.
@tswast This is one change I'm worried about. Do we rely on this non-versioned client for anything elsewhere that would care about getting shifted from v1beta1 to v1? It's going to require further changes to address if so.
There was a problem hiding this comment.
In our public samples, I always used the v1beta1 module. There are a few versions of pandas-gbq that use the versionless module, but I fixed that in the latest version. (googleapis/python-bigquery-pandas@e177978#diff-4db670026d33c02e5ad3dfbd5e4fd595R11)
So, yes: this will break a few versions of pandas-gbq. Might be worth making some aliases for BigQueryStorageClient and the necessary methods to avoid breakages.
There was a problem hiding this comment.
Given the proto messages have changed between v1beta1 and v1beta2 it seems you'd need more than just an alias to get the right client name. Things like tablereferences became a single string rather than a structured object, sharding strategy went away (everything is balanced), which makes converting a bit dicey if you were expecting the old strategy.
There was a problem hiding this comment.
Gotcha. When customers hit this, we can tell them to either upgrade pandas-gbq or pin their google-cloud-bigquery-storage version.
This PR contains both the changes to the synth config, as well as the generated output from running synth.
There's some work to try to accomodate v1alpha2 in this PR, but
there exists further oddities that must be tackled before we can
fully add generation of the alpha client here.
Also, note that this PR does not include manual client modifications
such as streaming offset resumption that are present in the v1beta1
client. Intent is to address further changes like that in subsequent
PRs.