-
Notifications
You must be signed in to change notification settings - Fork 15.6k
Add @task.kuberenetes_cmd
#46913
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add @task.kuberenetes_cmd
#46913
Conversation
4afbe38
to
f4567b7
Compare
Should I separate the test refactoring from this PR? |
6836426
to
da0a9c2
Compare
The taskflow tutorial docs change |
d0741ce
to
64269e3
Compare
I think that this should be adjusted to use TaskSDK and not depend on direct DB access, so made it a draft for now, before I would understand the proper way of doing it. |
6ca0c03
to
99943c9
Compare
Updated based on the recent example of I am still unsure: how to trigger system tests for example DAG in CI. |
b305d6b
to
683961a
Compare
683961a
to
a766622
Compare
Rebased it again - just to check - but I think this one will not get too much attention at least before RC1 - but please keep on rebasing it. Being on top of the main and fresh in the inbox helps. |
Yeah, I understand, there are a lot of things to do for the maintainers and committers now, and this PR is vast. Thank you! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Temporary block till we sort out the decorators in providers
a766622
to
07ae5c1
Compare
This should fix docstring and type-checking for decorator
f4000cf
to
0bae401
Compare
Adding a new task flow decorator to run generated cmds in
KubernetesPodOperator
.This decorator is based on the same idea as
@task.bash
: enrich Kubernetes pod command generation ability with simplified Python usage on the executor side to generate the command.With the
args_only
option, the decorator would set the return value as arguments of the image entrypoint instead of rewriting the cmds.This can significantly simplify usage for pure orchestrator cases and dynamic task mapping or generating the command from another task flow task result.
Changes:
@task.kubernetes_cmd
and register it inprovider.yaml
forcncf.kubernetes
;@task.kubernetes
unit tests to use the same base and have common tests with@task.kubernetes_cmd
;@overload
for@task.kubernetes
description in decorators pyi file to enable proper doc-string and signature checking in IDEscloses: #46414
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.