Skip to content

Conversation

@phanikumv
Copy link
Contributor

The BigQueryCreateEmptyDatasetOperator fails to execute with the below error when project_id is passed from a DAG, and not passed in connection.

[2022-08-26, 12:49:39 UTC] {taskinstance.py:1852} ERROR - Task failed with exception Traceback (most recent call last): File "/opt/airflow/airflow/providers/google/cloud/operators/bigquery.py", line 1369, in execute exists_ok=self.exists_ok, File "/opt/airflow/airflow/providers/google/common/hooks/base_google.py", line 463, in inner_wrapper return func(self, *args, **kwargs) File "/opt/airflow/airflow/providers/google/cloud/hooks/bigquery.py", line 461, in create_empty_dataset dataset_object = self.get_client(location=location).create_dataset( File "/opt/airflow/airflow/providers/google/cloud/hooks/bigquery.py", line 139, in get_client credentials=self.get_credentials(), File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 235, in __init__ _http=_http, File "/usr/local/lib/python3.7/site-packages/google/cloud/client/__init__.py", line 320, in __init__ _ClientProjectMixin.__init__(self, project=project, credentials=credentials) File "/usr/local/lib/python3.7/site-packages/google/cloud/client/__init__.py", line 268, in __init__ project = self._determine_default(project) File "/usr/local/lib/python3.7/site-packages/google/cloud/client/__init__.py", line 287, in _determine_default return _determine_default_project(project) File "/usr/local/lib/python3.7/site-packages/google/cloud/_helpers/__init__.py", line 152, in _determine_default_project _, project = google.auth.default() File "/usr/local/lib/python3.7/site-packages/google/auth/_default.py", line 616, in default raise exceptions.DefaultCredentialsError(_HELP_MESSAGE) google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application

This is because the get_client method call is currently not utilising the passed project_id while creating the authenticated client. This PR fixes this behaviour, and makes a change to accept project_id passed from the DAG or the connection, similar to how the project_id is passed for the other BigQuery operators


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@phanikumv phanikumv requested a review from turbaszek as a code owner August 26, 2022 13:15
@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Aug 26, 2022
@phanikumv phanikumv force-pushed the fix_google_operator branch from ba4584f to 5474a21 Compare August 26, 2022 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants