KPO Maintain backward compatibility for execute_complete and trigger run method #37454
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…run method In apache#37279 I introduce periodic logging of the container. During the process, I also changed a few event Dict key names and that is problematic for someone extending the KPO trigger. Also, the current execute_compelete method was unused in the KPO operator and was problematic if someone using it in an extended class since now the trigger can also emit an event even if the pod is in the pod intermediate state. one reported issue: apache#37279 (comment) In this PR I'm restoring the trigger event dict structure. Also, deprecating the execute_complete method
potiuk
approved these changes
Feb 15, 2024
33 tasks
pankajkoti
added a commit
to astronomer/airflow
that referenced
this pull request
Feb 18, 2024
I am observing an issue wrt to the recent deferrable KPO changes in PR apache#37279 and apache#37454, where when the pod fails to start within a specified timeout value, the KPO task is hanging forever whereas it is expected to fail after the timeout. This PR fixes the issue by correcting a logical error for detecting if elapsed timeout has occured for raising the timeout trigger event.
pankajastro
pushed a commit
that referenced
this pull request
Feb 18, 2024
…#37514) I am observing an issue wrt to the recent deferrable KPO changes in PR #37279 and #37454, where when the pod fails to start within a specified timeout value, the KPO task is hanging forever whereas it is expected to fail after the timeout. This PR fixes the issue by correcting a logical error for detecting if elapsed timeout has occured for raising the timeout trigger event.
1 task
16 tasks
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Jul 19, 2024
… (#37514) I am observing an issue wrt to the recent deferrable KPO changes in PR apache/airflow#37279 and apache/airflow#37454, where when the pod fails to start within a specified timeout value, the KPO task is hanging forever whereas it is expected to fail after the timeout. This PR fixes the issue by correcting a logical error for detecting if elapsed timeout has occured for raising the timeout trigger event. GitOrigin-RevId: 6412b06a7b35a0743656dd3b2160f390f40108c2
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Sep 20, 2024
… (#37514) I am observing an issue wrt to the recent deferrable KPO changes in PR apache/airflow#37279 and apache/airflow#37454, where when the pod fails to start within a specified timeout value, the KPO task is hanging forever whereas it is expected to fail after the timeout. This PR fixes the issue by correcting a logical error for detecting if elapsed timeout has occured for raising the timeout trigger event. GitOrigin-RevId: 6412b06a7b35a0743656dd3b2160f390f40108c2
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Nov 9, 2024
… (#37514) I am observing an issue wrt to the recent deferrable KPO changes in PR apache/airflow#37279 and apache/airflow#37454, where when the pod fails to start within a specified timeout value, the KPO task is hanging forever whereas it is expected to fail after the timeout. This PR fixes the issue by correcting a logical error for detecting if elapsed timeout has occured for raising the timeout trigger event. GitOrigin-RevId: 6412b06a7b35a0743656dd3b2160f390f40108c2
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
May 5, 2025
… (#37514) I am observing an issue wrt to the recent deferrable KPO changes in PR apache/airflow#37279 and apache/airflow#37454, where when the pod fails to start within a specified timeout value, the KPO task is hanging forever whereas it is expected to fail after the timeout. This PR fixes the issue by correcting a logical error for detecting if elapsed timeout has occured for raising the timeout trigger event. GitOrigin-RevId: 6412b06a7b35a0743656dd3b2160f390f40108c2
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
May 26, 2025
… (#37514) I am observing an issue wrt to the recent deferrable KPO changes in PR apache/airflow#37279 and apache/airflow#37454, where when the pod fails to start within a specified timeout value, the KPO task is hanging forever whereas it is expected to fail after the timeout. This PR fixes the issue by correcting a logical error for detecting if elapsed timeout has occured for raising the timeout trigger event. GitOrigin-RevId: 6412b06a7b35a0743656dd3b2160f390f40108c2
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Sep 21, 2025
… (#37514) I am observing an issue wrt to the recent deferrable KPO changes in PR apache/airflow#37279 and apache/airflow#37454, where when the pod fails to start within a specified timeout value, the KPO task is hanging forever whereas it is expected to fail after the timeout. This PR fixes the issue by correcting a logical error for detecting if elapsed timeout has occured for raising the timeout trigger event. GitOrigin-RevId: 6412b06a7b35a0743656dd3b2160f390f40108c2
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Oct 19, 2025
… (#37514) I am observing an issue wrt to the recent deferrable KPO changes in PR apache/airflow#37279 and apache/airflow#37454, where when the pod fails to start within a specified timeout value, the KPO task is hanging forever whereas it is expected to fail after the timeout. This PR fixes the issue by correcting a logical error for detecting if elapsed timeout has occured for raising the timeout trigger event. GitOrigin-RevId: 6412b06a7b35a0743656dd3b2160f390f40108c2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In #37279 I introduce periodic logging of the container.
During the process, I also changed a few event Dict key names
and that is problematic for someone extending the KPO trigger.
Also, the current execute_compelete method was unused in the KPO operator
and was problematic if someone using it in an extended class since
now the trigger can also emit an event even if the pod is in the pod intermediate state.
one reported issue: #37279 (comment)
In this PR I'm restoring the trigger event dict structure.
Also, deprecating the execute_complete method
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.