Skip to content

Add @task.kubernetes_cmd decorator to run returned command #46414

@insomnes

Description

@insomnes

Description

There is bash decorator which allows neat and simple bash command (returned str) execution. I want to add the same ability as kubernetes_cmd decorator. So we can configure the command we want to run inside specified image and not the python function.

Use case/motivation

Decorator flow is short and simple way to run airflow tasks, it's also superb in templating terms.
When you use airflow as orchestrator mostly to run commands in specific images it would be nice to have something similar to bash operator:

@task.bash
def also_run_this() -> str:
    return 'echo "ti_key={{ task_instance_key_str }}"'

also_this = also_run_this()

Which will allow us not to run some python function (as current @task.kubernetes do), but the command returned from function executed on airflow Scheduler / Worker side.

@task.kubernetes_cmd
def prepare_cmd() -> str:
    return 'python /path/inside/image/main.py --some-option={{ task_instance_key_str }}""'

I mean that prepare_cmd should be run inside executor / worker with access to airflow environment and this way we unleash full templating power of airflow for command generation in combination with python code without work around.
The generated string command is used as actual cmds argument for operator (we may decide to use list[str] as needed return value`.

This may simplify tons of KubernetesPodOperator usages in the described flow of container orchestration without the need to provide code on the airflow side.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions