feat(providers/spark): add post_submit_commands to SparkSubmitHook for sidecar lifecycle management#64391
Conversation
There was a problem hiding this comment.
I do not think it fixes the issue, we have fixed a similar issue differently, by adjusting the driver pod template and using k8s native sidecar containers
https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/
Which is supported in the k8s version airflow requires ( > 1.30)
|
The Could a maintainer please re-trigger the CI? Thanks!
I agree that using native Kubernetes sidecars (K8s 1.28+) is the much better architectural solution here. Unless you think this post_submit_commands parameter still has value for general worker-side teardown tasks (like local metric emission or temp file cleanup), I am happy to close this PR. |
There was a problem hiding this comment.
Pull request overview
Adds a post_submit_commands capability to the Apache Spark provider’s SparkSubmitHook to support running cleanup commands (e.g., service-mesh sidecar shutdown) after a Spark submission completes or is terminated, plus unit tests for the new behavior.
Changes:
- Add
post_submit_commandsparameter toSparkSubmitHookand persist it on the hook instance. - Implement
_run_post_submit_commands()to execute configured commands aftersubmit()andon_kill(). - Add new unit tests covering command execution, resilience to failures/timeouts, and basic integration points.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py | Introduces post_submit_commands plumbing and executes post-submit shell commands from submit() and on_kill(). |
| providers/apache/spark/tests/unit/apache/spark/hooks/test_spark_submit_post_commands.py | Adds unit tests for post-submit command behavior and integration with submit() / on_kill(). |
85058fb to
af8c125
Compare
f93d5d1 to
7ce0c62
Compare
|
Again - static checks fail . |
7ce0c62 to
49d54d6
Compare
…r sidecar lifecycle management
49d54d6 to
03847ba
Compare
fixed, all checks are passed now. |
Summary
Closes #50958
When running Spark jobs on Kubernetes with a service mesh like Istio,
the sidecar proxy container does not automatically exit when the Spark
driver finishes. This leaves the pod in a running state indefinitely,
blocking job completion in Airflow.
This PR adds a
post_submit_commandsparameter toSparkSubmitHookthat runs a list of shell commands after the Spark job finishes —
whether it completes successfully or is killed via
on_kill().Example usage:
Changes
post_submit_commands: list[str] | None = Noneparameter toSparkSubmitHook.__init__()_run_post_submit_commands()method that runs each command viasubprocess.run(shell=True)with a 30s timeout, logging output andnon-zero exit codes as warnings (never raises)
submit()(after job completion) andon_kill()(after process termination)
backward compatibility