-
Notifications
You must be signed in to change notification settings - Fork 15.6k
Run Task failure callbacks on DAG Processor when task is externally killed #53058
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor nits, looks good otherwise.
fb0fa28
to
9d562d0
Compare
…illed Until apache#44354 is implemented, tasks killed externally or when supervisor process dies unexpectedly, users have no way of knowing this happened. This has been a blocker for Airflow 3.0 adoption for some: - apache#44354 - https://apache-airflow.slack.com/archives/C07813CNKA8/p1751057525231389 apache#44354 is more involved and we might not get to it for Airflow 3.1 -- so this is a good fix until then similar to how we run Dag Run callback.
9d562d0
to
2f67859
Compare
Co-authored-by: Ash Berlin-Taylor <[email protected]>
Backport failed to create: v3-0-test. View the failure log Run details
You can attempt to backport this manually by running: cherry_picker a5211f2 v3-0-test This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continue |
…illed (apache#53058) Until apache#44354 is implemented, tasks killed externally or when supervisor process dies unexpectedly, users have no way of knowing this happened. This has been a blocker for Airflow 3.0 adoption for some: - apache#44354 - https://apache-airflow.slack.com/archives/C07813CNKA8/p1751057525231389 apache#44354 is more involved and we might not get to it for Airflow 3.1 -- so this is a good fix until then similar to how we run Dag Run callback. (cherry-picked from a5211f2)
Backport PR: #53143 |
…illed (apache#53058) Until apache#44354 is implemented, tasks killed externally or when supervisor process dies unexpectedly, users have no way of knowing this happened. This has been a blocker for Airflow 3.0 adoption for some: - apache#44354 - https://apache-airflow.slack.com/archives/C07813CNKA8/p1751057525231389 apache#44354 is more involved and we might not get to it for Airflow 3.1 -- so this is a good fix until then similar to how we run Dag Run callback.
…illed (#53058) (#53143) Until #44354 is implemented, tasks killed externally or when supervisor process dies unexpectedly, users have no way of knowing this happened. This has been a blocker for Airflow 3.0 adoption for some: - #44354 - https://apache-airflow.slack.com/archives/C07813CNKA8/p1751057525231389 #44354 is more involved and we might not get to it for Airflow 3.1 -- so this is a good fix until then similar to how we run Dag Run callback. (cherry-picked from a5211f2)
…illed (#53058) (#53143) Until #44354 is implemented, tasks killed externally or when supervisor process dies unexpectedly, users have no way of knowing this happened. This has been a blocker for Airflow 3.0 adoption for some: - #44354 - https://apache-airflow.slack.com/archives/C07813CNKA8/p1751057525231389 #44354 is more involved and we might not get to it for Airflow 3.1 -- so this is a good fix until then similar to how we run Dag Run callback. (cherry-picked from a5211f2)
…illed (#53058) (#53143) Until #44354 is implemented, tasks killed externally or when supervisor process dies unexpectedly, users have no way of knowing this happened. This has been a blocker for Airflow 3.0 adoption for some: - #44354 - https://apache-airflow.slack.com/archives/C07813CNKA8/p1751057525231389 #44354 is more involved and we might not get to it for Airflow 3.1 -- so this is a good fix until then similar to how we run Dag Run callback. (cherry-picked from a5211f2)
…illed (apache#53058) Until apache#44354 is implemented, tasks killed externally or when supervisor process dies unexpectedly, users have no way of knowing this happened. This has been a blocker for Airflow 3.0 adoption for some: - apache#44354 - https://apache-airflow.slack.com/archives/C07813CNKA8/p1751057525231389 apache#44354 is more involved and we might not get to it for Airflow 3.1 -- so this is a good fix until then similar to how we run Dag Run callback.
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to #53058 Fixes #52824 Fixes #51402 Closes #51949 Related to #53654 Related to #53618
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to #53058 Fixes #52824 Fixes #51402 Closes #51949 Related to #53654 Related to #53618 (cherry picked from commit ef80507)
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
Until #44354 is implemented, tasks killed externally or when supervisor process dies unexpectedly, users have no way of knowing this happened.
This has been a blocker for Airflow 3.0 adoption for some:
#44354 is more involved and we might not get to it for Airflow 3.1 -- so this is a good fix until then similar to how we run Dag Run callback.
Ran a example DAG:
DAG Processor Logs after I killed the Supervisor & task process when the task is running:
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in airflow-core/newsfragments.