-
Notifications
You must be signed in to change notification settings - Fork 15.6k
Description
Description
There is bash decorator which allows neat and simple bash command (returned str) execution. I want to add the same ability as kubernetes_cmd
decorator. So we can configure the command we want to run inside specified image and not the python function.
Use case/motivation
Decorator flow is short and simple way to run airflow tasks, it's also superb in templating terms.
When you use airflow as orchestrator mostly to run commands in specific images it would be nice to have something similar to bash operator:
@task.bash
def also_run_this() -> str:
return 'echo "ti_key={{ task_instance_key_str }}"'
also_this = also_run_this()
Which will allow us not to run some python function (as current @task.kubernetes
do), but the command returned from function executed on airflow Scheduler / Worker side.
@task.kubernetes_cmd
def prepare_cmd() -> str:
return 'python /path/inside/image/main.py --some-option={{ task_instance_key_str }}""'
I mean that prepare_cmd
should be run inside executor / worker with access to airflow environment and this way we unleash full templating power of airflow for command generation in combination with python code without work around.
The generated string command is used as actual cmds argument for operator (we may decide to use list[str]
as needed return value`.
This may simplify tons of KubernetesPodOperator
usages in the described flow of container orchestration without the need to provide code on the airflow side.
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct