Skip to content

Commit 28d1bf8

Browse files
lwyszomiLukasz Wyszomirskimoiseenkovkosteevahidalgob
authored
Update SDKs for google provider package (#30067)
* Update SDK versions for Google provider * Adjust google ads operators to v12 Changes: - fix tests/system/providers/google/cloud/bigquery/example_bigquery_queries.py - fix tests/system/providers/google/cloud/bigquery/example_bigquery_queries_async.py * Fix GCS system tests * Fix CloudBuild unit test * Update BigTable operators to accomodate for new dependencies. * Fix Cloud Tasks System tests Tasks dag was quite flaky without the retry option in the run_task step, but it's consistently green with the option set. We also add a GCP_APP_ENGINE_LOCATION env variable since this depends on the used GCP Project App Engine's location * Add setup docstring to Tasks system tests. * Update Vision operators to accommodate new dependencies. Changes: - fix methods for CloudVisionHook - fix Vision Operators - fix tests/providers/google/cloud/hooks/test_vision.py - fix tests/providers/google/cloud/operators/test_vision.py - fix tests/system/providers/google/cloud/vision/example_vision_annotate_image.py - fix tests/system/providers/google/cloud/vision/example_vision_autogenerated.py - fix tests/system/providers/google/cloud/vision/example_vision_explicit.py * Update SpeechToText operators to accommodate new dependencies. Changes: - fix synthesize_speech method for CloudTextToSpeechHook - fix CloudSpeechToTextRecognizeSpeechOperator - fix tests/providers/google/cloud/operators/test_speech_to_text.py - fix tests/providers/google/cloud/hooks/test_text_to_speech.py - fix tests/providers/google/cloud/hooks/test_speech_to_text.py * Update Translate Speech operators to accommodate new dependencies. Changes: - fix synthesize_speech method for CloudTextToSpeechHook - fix CloudTranslateSpeechOperator - tests/providers/google/cloud/operators/test_translate_speech.py * Update VideoIntelligence operators to accommodate new dependencies. Changes: - fix annotate_video method for CloudVideoIntelligenceHook - fix VideoIntelligence Operators - fix tests/providers/google/cloud/hooks/test_video_intelligence.py - fix tests/providers/google/cloud/operators/test_video_intelligence.py * Update Compute Engine operators to accomodate for new dependencies. Changes: - added wait_for_operation_complete() method to check the execution flow - added new attribute cmd_timeout for ComputeEngineSSHHook * Fix Stackdriver system test This test has not worked because of slack channel and credentials not being setup. We now test the same operators by creating notification channels and policy alerts against pubsub topics, which don't need to exist before the test is ran, making the test self-contained. * Update Natural Language operators to accommodate new dependencies. Changes: - fix airflow/providers/google/cloud/operators/natural_language.py - fix airflow/providers/google/cloud/hooks/natural_language.py - fix tests/providers/google/cloud/hooks/test_natural_language.py - fix tests/providers/google/cloud/operators/test_natural_language.py - fix tests/system/providers/google/cloud/natural_language/example_natural_language.py * Update Composer system tests. Fix environment id to contain underscores. * Update AutoML operators to accommodate new dependencies. Changes: - add timeout parameter to all long-running operations for operators - fix tests/system/providers/google/cloud/automl/example_automl_dataset.py - fix tests/system/providers/google/cloud/automl/example_automl_model.py - fix tests/system/providers/google/cloud/automl/example_automl_nl_text_extraction.py - fix tests/system/providers/google/cloud/automl/example_automl_vision_classification.py * Fix Cloud SQL delete operator For some delete instance operations, the operation stops being available ~9 seconds after completion, so we need a shorter sleep time to make sure we don'tmiss the DONE status. * Update VertexAI operators to accommodate new dependencies. * Add SQL to Sheets Test instructions * Update Dataproc Metastore operators to accommodate new dependencies. * Update Dataproc operators to accommodate new dependencies. * Update Dataflow sys tests to new sdk * Update Dataproc on gke operators to accommodate new dependencies. * Update MLEngine operators to accomodate new dependencies. Changes: - update train model that is used for prediction - update version and runner for ApacheBeam in utils for MLEngine - update connection inside async hook * Update Dataprep operators to accommodate new dependencies. Changes: - fix tests/system/providers/google/cloud/dataprep/example_dataprep.py * Add Dataflow Go system test * Update providers.yaml for google * fixup! Update providers.yaml for google * Google SDK Fixes after rebase - Secret Manager was missing updating to v2, now expects a request dict - Compute ssh had a bug when no cmd_timeout was passed - Cloud Build tests were improved/refactored in community, so deleting old ones - googleapiclient.errors.HttpError was incorrectly used in our tests, it it didn´t matter before but a change in the class makes HttpError() raise an error in initialization the way we were using it before - fix static checks * Fix Google providers type errors --------- Co-authored-by: Lukasz Wyszomirski <wyszomirski@google.com> Co-authored-by: Maksim Moiseenkov <maksim_moiseenkov@epam.com> Co-authored-by: Eugene Kostieiev <kosteev@google.com> Co-authored-by: Augusto Hidalgo <augustoh@google.com> Co-authored-by: Beata Kossakowska <bkossakowska@google.com> Co-authored-by: Ulada Zakharava <uladaz@google.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
1 parent a03f6cc commit 28d1bf8

File tree

118 files changed

+1381
-837
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

118 files changed

+1381
-837
lines changed

‎airflow/providers/apache/beam/provider.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ versions:
4343

4444
dependencies:
4545
- apache-airflow>=2.4.0
46-
- apache-beam>=2.33.0
46+
- apache-beam>=2.47.0
4747

4848
integrations:
4949
- integration-name: Apache Beam

‎airflow/providers/google/ads/hooks/ads.py‎

Lines changed: 7 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -21,26 +21,18 @@
2121
from tempfile import NamedTemporaryFile
2222
from typing import IO, Any
2323

24+
from google.ads.googleads.client import GoogleAdsClient
25+
from google.ads.googleads.errors import GoogleAdsException
26+
from google.ads.googleads.v12.services.services.customer_service import CustomerServiceClient
27+
from google.ads.googleads.v12.services.services.google_ads_service import GoogleAdsServiceClient
28+
from google.ads.googleads.v12.services.types.google_ads_service import GoogleAdsRow
29+
from google.api_core.page_iterator import GRPCIterator
2430
from google.auth.exceptions import GoogleAuthError
2531

2632
from airflow import AirflowException
2733
from airflow.compat.functools import cached_property
2834
from airflow.hooks.base import BaseHook
2935
from airflow.providers.google.common.hooks.base_google import get_field
30-
from airflow.providers.google_vendor.googleads.client import GoogleAdsClient
31-
from airflow.providers.google_vendor.googleads.errors import GoogleAdsException
32-
from airflow.providers.google_vendor.googleads.v12.services.services.customer_service import (
33-
CustomerServiceClient,
34-
)
35-
from airflow.providers.google_vendor.googleads.v12.services.services.google_ads_service import (
36-
GoogleAdsServiceClient,
37-
)
38-
from airflow.providers.google_vendor.googleads.v12.services.services.google_ads_service.pagers import (
39-
SearchPager,
40-
)
41-
from airflow.providers.google_vendor.googleads.v12.services.types.google_ads_service import (
42-
GoogleAdsRow,
43-
)
4436

4537

4638
class GoogleAdsHook(BaseHook):
@@ -238,7 +230,7 @@ def _search(
238230

239231
return self._extract_rows(iterators)
240232

241-
def _extract_rows(self, iterators: list[SearchPager]) -> list[GoogleAdsRow]:
233+
def _extract_rows(self, iterators: list[GRPCIterator]) -> list[GoogleAdsRow]:
242234
"""
243235
Convert Google Page Iterator (GRPCIterator) objects to Google Ads Rows
244236

‎airflow/providers/google/cloud/_internal_client/secret_manager_client.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ def get_secret(self, secret_id: str, project_id: str, secret_version: str = "lat
7171
"""
7272
name = self.client.secret_version_path(project_id, secret_id, secret_version)
7373
try:
74-
response = self.client.access_secret_version(name)
74+
response = self.client.access_secret_version(request={"name": name})
7575
value = response.payload.data.decode("UTF-8")
7676
return value
7777
except NotFound:

‎airflow/providers/google/cloud/example_dags/example_compute.py‎

Lines changed: 0 additions & 107 deletions
This file was deleted.

‎airflow/providers/google/cloud/example_dags/example_compute_ssh.py‎

Lines changed: 0 additions & 90 deletions
This file was deleted.

‎airflow/providers/google/cloud/hooks/automl.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@
4848
)
4949
from google.protobuf.field_mask_pb2 import FieldMask
5050

51+
from airflow import AirflowException
5152
from airflow.compat.functools import cached_property
5253
from airflow.providers.google.common.consts import CLIENT_INFO
5354
from airflow.providers.google.common.hooks.base_google import PROVIDE_PROJECT_ID, GoogleBaseHook
@@ -93,6 +94,14 @@ def get_conn(self) -> AutoMlClient:
9394
self._client = AutoMlClient(credentials=self.get_credentials(), client_info=CLIENT_INFO)
9495
return self._client
9596

97+
def wait_for_operation(self, operation: Operation, timeout: float | None = None):
98+
"""Waits for long-lasting operation to complete."""
99+
try:
100+
return operation.result(timeout=timeout)
101+
except Exception:
102+
error = operation.exception(timeout=timeout)
103+
raise AirflowException(error)
104+
96105
@cached_property
97106
def prediction_client(self) -> PredictionServiceClient:
98107
"""

0 commit comments

Comments
 (0)