Yandex dataproc deduce default service account #35059

Piatachock · 2023-10-19T14:18:00Z

New features for YandexCloud Provider:

Support User-Agent header parametrization via environent variable
Store Service Account ID from Connection in Hook; use it as default in DataprocClusterCreateOperator

Taragolis · 2023-10-19T14:47:10Z

airflow/providers/yandex/hooks/yandex.py

Maybe better make it as a part of Provider Configuration?

Setting Configuration Options

Global Configuration Reference

OpenLinage Configuration Reference

eladkal · 2023-10-22T05:50:10Z

Static checks are failing

airflow/providers/yandex/provider.yaml

Taragolis · 2023-10-25T00:48:04Z

airflow/providers/yandex/hooks/yandex.py

Suggested change

conf.get("yandex", "sdk_user_agent_prefix"),

conf.get("yandex", "sdk_user_agent_prefix", fallback=""),

Piatachock · 2023-10-25T08:50:43Z

Static checks are failing

Sorry, forgot to mark this PR as WIP.
I'm actually trying to build provider package locally to check it in our Managed Airflow installation end-to-end.
Do you, by any chance, know where should I look for any info on the subject? I've found dev folder and relevant scripts, but still struggling atm.

Taragolis · 2023-10-25T09:24:39Z

it in our Managed Airflow

BTW, you or some one from the Yandex also could add link to Yandex Managed Service for Apache Airflow into Airflow Ecosystem, this placed into another repo, however changes could be done by simple click on Suggest a change on this page and make PR changes into the Github Editor

The only one thing that I found that International Link to Yandex's managed service https://cloud.yandex.com/en/services/managed-airflow right now return 404

Taragolis · 2023-10-25T09:59:15Z

I'm actually trying to build provider package locally to check it in our Managed Airflow installation end-to-end.
Do you, by any chance, know where should I look for any info on the subject? I've found dev folder and relevant scripts, but still struggling atm.

You could run this breeze command

breeze release-management prepare-provider-packages yandex --version-suffix-for-pypi dev0 --package-format both

This would create two distribution (wheel, source) in dist directory

Good version of Docker: 24.0.6.
Good version of docker-compose: 2.22.0

...

Prepared provider package yandex in format both
===================================================================================

Summary of prepared packages:

    Prepared:
yandex

===================================================================================
apache-airflow-providers-yandex-3.5.0.dev0.tar.gz
apache_airflow_providers_yandex-3.5.0.dev0-py3-none-any.whl
===================================================================================

All good! Airflow packages are prepared in dist folder

… use it in DataprocCreateClusterOperator

…v variable

Piatachock · 2023-10-31T12:56:47Z

Thanks! Managed to build provider locally and test it on our managed solution. Works as expected
This PR is ready for review

Taragolis · 2023-11-01T09:35:23Z

Need to add test for this changes.

For validate configuration, you might use conf_vars - context manager / decorator. Some samples:

airflow/tests/always/test_secrets_local_filesystem.py

Lines 431 to 446 in eb450b9

    
               @conf_vars( 
        
                   { 
        
                       ( 
        
                           "secrets", 
        
                           "backend", 
        
                       ): "airflow.secrets.local_filesystem.LocalFilesystemBackend", 
        
                       ("secrets", "backend_kwargs"): '{"variables_file_path": "var.env"}', 
        
                   } 
        
               ) 
        
               def test_load_secret_backend_LocalFilesystemBackend(self): 
        
                   with mock_local_file("KEY_A=VAL_A"): 
        
                       backends = ensure_secrets_loaded() 
        
                       backend_classes = [backend.__class__.__name__ for backend in backends] 
        
                       assert "LocalFilesystemBackend" in backend_classes 
        
                       assert Variable.get("KEY_A") == "VAL_A"

airflow/tests/jobs/test_base_job.py

Lines 113 to 123 in 1aa91a4

    
               @pytest.mark.parametrize( 
        
                   "job_runner, job_type,job_heartbeat_sec", 
        
                   [(SchedulerJobRunner, "scheduler", "11"), (TriggererJobRunner, "triggerer", "9")], 
        
               ) 
        
               def test_heart_rate_via_constructor_persists(self, job_runner, job_type, job_heartbeat_sec): 
        
                   """Ensure heartrate passed via constructor is set correctly""" 
        
                   with conf_vars({(job_type.lower(), "job_heartbeat_sec"): job_heartbeat_sec}): 
        
                       job = Job(heartrate=12) 
        
                       job_runner(job) 
        
                       # heartrate should be 12 since we passed that to the constructor directly 
        
                       assert job.heartrate == 12

Piatachock · 2023-11-03T09:28:04Z

Andrey, thanks for your approval! Is there anything else that should be done prior to merge this PR?

potiuk · 2023-11-03T15:52:20Z

Nope. It looks cool.

potiuk · 2023-11-03T15:58:23Z

Good to see this "per-provider" configuration feature... But that reminds me that we need a bit more "discovery"

Look at "docs/apache-airflow-providers-celery" - there is configurtions-ref.rst which you have to add to yandex and you will have to also add configuration to index.rst @Piatachock also you should add link to https://github.com/apache/airflow/blob/main/docs/apache-airflow/configurations-ref.rst

That reminds me that we should add pre-commit and automation - this all should happen semi-automatically or at least should be verified by pre-commit (but it does not yet).

The change apache#35059 missed configuration docs but it did not fail the test suite. We need to improve it in the future, but for now just adding missing configuration documentations should fix doc build failures.

The change #35059 missed configuration docs but it did not fail the test suite. We need to improve it in the future, but for now just adding missing configuration documentations should fix doc build failures.

--------- Co-authored-by: Petr Reznikov <prez@yandex-team.ru>

The change apache#35059 missed configuration docs but it did not fail the test suite. We need to improve it in the future, but for now just adding missing configuration documentations should fix doc build failures.

boring-cyborg bot added area:providers provider:yandex labels Oct 19, 2023

Taragolis reviewed Oct 19, 2023

View reviewed changes

Piatachock force-pushed the yandex_dataproc_deduce_default_service_account_id branch from 9d393da to 1d44ec6 Compare October 19, 2023 18:06

Taragolis reviewed Oct 25, 2023

View reviewed changes

providers/yandex : deduce default service_account_id from connection,…

d6deba4

… use it in DataprocCreateClusterOperator

Piatachock force-pushed the yandex_dataproc_deduce_default_service_account_id branch from 1d44ec6 to c1e8f4f Compare October 31, 2023 08:49

providers/yandex : allow parametrization of airflow user-agent via en…

a716ec5

…v variable

Piatachock force-pushed the yandex_dataproc_deduce_default_service_account_id branch from c1e8f4f to a716ec5 Compare October 31, 2023 10:48

Piatachock force-pushed the yandex_dataproc_deduce_default_service_account_id branch 2 times, most recently from 31ebfbc to 3dd271c Compare November 1, 2023 07:52

Add unit test to check SDK user-agent prefix

2eddb69

Piatachock force-pushed the yandex_dataproc_deduce_default_service_account_id branch from 3dd271c to 2eddb69 Compare November 1, 2023 13:27

Taragolis approved these changes Nov 1, 2023

View reviewed changes

potiuk approved these changes Nov 3, 2023

View reviewed changes

potiuk merged commit 0b850a9 into apache:main Nov 3, 2023

potiuk mentioned this pull request Nov 3, 2023

Add configuration files for yandex #35420

Merged

eladkal mentioned this pull request Nov 8, 2023

Status of testing Providers that were prepared on November 08, 2023 #35540

Closed

60 tasks

romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Nov 10, 2023

Yandex dataproc deduce default service account (apache#35059)

4c54676

--------- Co-authored-by: Petr Reznikov <prez@yandex-team.ru>

	conf.get("yandex", "sdk_user_agent_prefix"),
	conf.get("yandex", "sdk_user_agent_prefix", fallback=""),

Yandex dataproc deduce default service account #35059

Yandex dataproc deduce default service account #35059

Uh oh!

Conversation

Piatachock commented Oct 19, 2023

Uh oh!

Taragolis Oct 19, 2023

Choose a reason for hiding this comment

Uh oh!

eladkal commented Oct 22, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Taragolis Oct 25, 2023

Choose a reason for hiding this comment

Uh oh!

Piatachock commented Oct 25, 2023

Uh oh!

Taragolis commented Oct 25, 2023

Uh oh!

Taragolis commented Oct 25, 2023

Uh oh!

Piatachock commented Oct 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Taragolis commented Nov 1, 2023

Uh oh!

Piatachock commented Nov 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Nov 3, 2023

Uh oh!

potiuk commented Nov 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Piatachock commented Oct 31, 2023 •

edited

Loading

Piatachock commented Nov 3, 2023 •

edited

Loading