Add extra validation to SpannerToBigQuery Template#2035
Merged
jrmccluskey merged 3 commits intoGoogleCloudPlatform:mainfrom Dec 3, 2024
Merged
Add extra validation to SpannerToBigQuery Template#2035jrmccluskey merged 3 commits intoGoogleCloudPlatform:mainfrom
jrmccluskey merged 3 commits intoGoogleCloudPlatform:mainfrom
Conversation
Contributor
|
Sorry for late reply.
Would you mind pointing out the code path for this Exception? I'm wondering by any chance the original setting could work in some use case, e.g. CREATE_IF_NEEDED + not provide schema + the table actually existed, and with this change it now fails. |
Contributor
Author
|
The failure pops up here: |
Contributor
~I see, NPE raised in getGcsFileAsString. Currently the error will happen regardless of CREATE_DISPOSITION. We should move
never mind, I see it already did in writeToBigQuery method |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SpannerToBigQuery template wasn't failing with a clear error if the create disposition was not CREATE_NEVER but no BigQuery schema was provided, instead just throwing a vague NullPointerException in the write step. This replaces that exception with an IllegalArgumentException and a clear message before runtime.