/gcbrun

/gcbrun

/gcbrun

/gcbrun

Thanks for this David. Getting this into a release that resolves #1398 would be really helpful for us. Although, I'm not sure how this PR plays in with https://github.com/GoogleCloudDataproc/spark-bigquery-connector/pull/1345/files which seems really nice too. If we could avoid the 'too large' problem and also not have to provide the materialization dataset that would be a big win.

Hi @kristopherkane

Thanks for this David. Getting this into a release that resolves #1398 would be really helpful for us. Although, I'm not sure how this PR plays in with https://github.com/GoogleCloudDataproc/spark-bigquery-connector/pull/1345/files which seems really nice too. If we could avoid the 'too large' problem and also not have to provide the materialization dataset that would be a big win.

Thanks, the idea is to make materializationDataset optional for users who need it (due to permissions issues, for example), but most users would be able to avoid it.

/gcbrun

/gcbrun