Add documentation generation for CLI commands from executors #33081
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
f6ae439 to
7d581ff
Compare
Member
Author
|
Example Celery docs:
|
Member
Author
|
I will need to add a second one to add links to the providers from Airflow docs - it will have to be done after this one is merged though due to cyclic celery -> airflow -> kubernetes -> celery dependency and I have to make sure the inventory for the Celery / Kubernetes providers is pushed to S3 before the Airflow one might succed. |
o-nikolas
reviewed
Aug 3, 2023
airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py
Outdated
Show resolved
Hide resolved
7d581ff to
c44ae7b
Compare
c44ae7b to
3e5b76e
Compare
The apache#29055 moved relevant CLI command definition to executors but it automatically removed the command from generated documentation. This PR brings it back for both Celery and Cncf Kubernetes provider Closes: apache#27932
3e5b76e to
3d3f80a
Compare
Member
Author
|
That one should be ready - for now it has only "providers" side - I will add separately the "airflow" side to link to the documentation of the CLI commands in providers cc: @eladkal |
Member
Author
|
All green ! |
44 tasks
o-nikolas
reviewed
Aug 4, 2023
| :meta private: | ||
| """ | ||
| return CeleryExecutor._get_parser() |
Contributor
There was a problem hiding this comment.
Very nice now abstracted away to the base class!
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Aug 5, 2023
The PROD image verification happens after the images are pulled in the "wait for PROD images" step. This verification is pretty helpful in detecting cases where there are some "installed airflow" problems (for example recently it helped to avoid a circular import problem in apache#33081 as one of the tests failed when images were verified. However PROD image wait might fail for other reasons and sometimes might be "neglected" as temporary failure. Separating verification will allow to clearly surface that the problem is with verification, not pulling the images.
potiuk
added a commit
that referenced
this pull request
Aug 5, 2023
The PROD image verification happens after the images are pulled in the "wait for PROD images" step. This verification is pretty helpful in detecting cases where there are some "installed airflow" problems (for example recently it helped to avoid a circular import problem in #33081 as one of the tests failed when images were verified. However PROD image wait might fail for other reasons and sometimes might be "neglected" as temporary failure. Separating verification will allow to clearly surface that the problem is with verification, not pulling the images.
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Aug 5, 2023
The apache#33081 added documentation generation for CLI commands contributed by executors coming from providers. Adding Airflow documentation in the same PR was not really possible because it created a documentation reference circle and we had to wait until the documentation inventory has been released in S3 and since this already happened, we can now add links to provider CLIs from Airflow.
potiuk
added a commit
that referenced
this pull request
Aug 5, 2023
…33143) The #33081 added documentation generation for CLI commands contributed by executors coming from providers. Adding Airflow documentation in the same PR was not really possible because it created a documentation reference circle and we had to wait until the documentation inventory has been released in S3 and since this already happened, we can now add links to provider CLIs from Airflow.
ephraimbuddy
pushed a commit
that referenced
this pull request
Aug 8, 2023
…33143) The #33081 added documentation generation for CLI commands contributed by executors coming from providers. Adding Airflow documentation in the same PR was not really possible because it created a documentation reference circle and we had to wait until the documentation inventory has been released in S3 and since this already happened, we can now add links to provider CLIs from Airflow. (cherry picked from commit 9910d92)
ephraimbuddy
pushed a commit
that referenced
this pull request
Aug 8, 2023
The PROD image verification happens after the images are pulled in the "wait for PROD images" step. This verification is pretty helpful in detecting cases where there are some "installed airflow" problems (for example recently it helped to avoid a circular import problem in #33081 as one of the tests failed when images were verified. However PROD image wait might fail for other reasons and sometimes might be "neglected" as temporary failure. Separating verification will allow to clearly surface that the problem is with verification, not pulling the images. (cherry picked from commit c310703)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The #29055 moved relevant CLI command definition to executors but it automatically removed the command from generated documentation.
Closes: #27932
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.