Fix lingering task supervisors when EOF is missed
#51180
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
closes apache#50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly.
2 tasks
amoghrajesh
approved these changes
May 29, 2025
Co-authored-by: Amogh Desai <[email protected]>
kaxil
commented
May 29, 2025
sanederchik
pushed a commit
to sanederchik/airflow
that referenced
this pull request
Jun 7, 2025
closes apache#50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly.
ashb
pushed a commit
that referenced
this pull request
Jun 20, 2025
closes #50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly. (cherry picked from commit a2651f1)
github-actions bot
pushed a commit
that referenced
this pull request
Jun 20, 2025
…1180) closes #50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly. (cherry picked from commit a2651f1) Co-authored-by: Kaxil Naik <[email protected]>
Backport successfully created: v3-0-test
|
jedcunningham
pushed a commit
that referenced
this pull request
Jun 20, 2025
…1180) (#51970) closes #50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly. (cherry picked from commit a2651f1) Co-authored-by: Kaxil Naik <[email protected]>
jose-lehmkuhl
pushed a commit
to jose-lehmkuhl/airflow
that referenced
this pull request
Jul 11, 2025
closes apache#50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #50500
Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF.
While the supervisor "shouldn't" get stuck, the reality is that under extreme load the selector can miss final EOF events - which might have happened previously under high CPU load + the bug in #51023. This adds a safety net to prevent resource leaks and hanging processes (Pods accumulate for CE & KE). The timeout is generous (60s) to avoid premature cleanup of legitimate slow socket operations. I tried reproducing it reliably after my first attempt in #51020 but haven't been able to reproduce it.
Changes
workers.socket_cleanup_timeoutconfig (default 60s) to force-close stuck sockets after task process exits^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.