Skip to content

Fix provider.yaml errors due to exit(0) in test#17858

Merged
potiuk merged 1 commit intoapache:mainfrom
potiuk:fix-wrong-provider-yaml-verification
Aug 27, 2021
Merged

Fix provider.yaml errors due to exit(0) in test#17858
potiuk merged 1 commit intoapache:mainfrom
potiuk:fix-wrong-provider-yaml-verification

Conversation

@potiuk
Copy link
Copy Markdown
Member

@potiuk potiuk commented Aug 26, 2021

There were a few errors in provider structure resulting from
accidental exit(0) in provider's tests. Luckily none of the changes
have been released yet.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@potiuk potiuk requested a review from ashb as a code owner August 26, 2021 20:16
@potiuk potiuk requested review from eladkal, mik-laj and uranusjr August 26, 2021 20:16
@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented Aug 26, 2021

I found some errors in provider.yaml while preparing the new issue templates. Turned out that there was a 'sys.exit(0)' in one of the tests (for duplicate integration name) and it "SUCCEEDED" rather than failed, ignoring the rest of the tests.

Luckily I caught it before release - but the change is more than one line - I had to move the new local_to_s3 transfer to operators (because it was not a transfer at all). Transfers require two hooks from different providers and this is not the case here.

@eladkal
Copy link
Copy Markdown
Contributor

eladkal commented Aug 26, 2021

Transfers require two hooks from different providers and this is not the case here.

How so? There is LocalFilesystemToGCSOperator in transfers of google provider
https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/local_to_gcs.py
it uses only GCSHook.

My suggestion in the issue itself was to do the same as GCP:
#17168

Same with Azure:
https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/transfers/local_to_adls.py
https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/transfers/file_to_wasb.py

@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented Aug 27, 2021

How so? There is LocalFilesystemToGCSOperator in transfers of google provider

Ah Indeed @kaxil @eladkal . I missed the "Local" Integration. Shame on me.

There were a few errors in provider structure resulting from
accidental exit(0) in provider's tests. Luckily none of the changes
have been released yet.
@potiuk potiuk force-pushed the fix-wrong-provider-yaml-verification branch from f02a34d to 7f1ad08 Compare August 27, 2021 08:30
@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented Aug 27, 2021

Should be fixed now @eladkal @kaxil! Thanks for being watchful!

@github-actions
Copy link
Copy Markdown

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Aug 27, 2021
@potiuk potiuk merged commit 41632e0 into apache:main Aug 27, 2021
@potiuk potiuk deleted the fix-wrong-provider-yaml-verification branch August 27, 2021 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:providers full tests needed We need to run full set of tests for this PR to merge provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants