Hashicorp Vault Secrets Backend not used in PythonVirtualenvOperator

[Enucatl]

Apache Airflow Provider(s)

hashicorp

Versions of Apache Airflow Providers

FROM docker.io/apache/airflow:3.0.4@sha256:6c07868196d1880d956f4b5ddf50244bc281faab39704a8940b1c1f3159a145e

Apache Airflow version

3.0.4

Operating System

Ubuntu 22.04.5 LTS

Deployment

Docker-Compose

Deployment details

Full docker-compose if needed: https://github.com/Enucatl/docker-airflow/blob/main/docker-compose.yml

What happened

Connections defined in Hashicorp Vault are not available within PythonVirtualenvOperators

What you think should happen instead

test_connection log (no virtualenv = successful)

Log message source details: sources=["/opt/airflow/logs/dag_id=test_vault_virtualenv/run_id=scheduled__2025-08-17T00:00:00+00:00/task_id=test_connection/attempt=1.log"]
::group::Log message source details: sources=["/opt/airflow/logs/dag_id=test_vault_virtualenv/run_id=scheduled__2025-08-17T00:00:00+00:00/task_id=test_connection/attempt=1.log"]
[2025-08-19, 08:42:12] INFO - DAG bundles loaded: dags-folder: source="airflow.dag_processing.bundles.manager.DagBundlesManager"
[2025-08-19, 08:42:12] INFO - Filling up the DagBag from /opt/airflow/dags/test_vault_virtualenv.py: source="airflow.models.dagbag.DagBag"
[2025-08-19, 08:42:12] INFO - Connection Retrieved 'stva': source="airflow.hooks.base"
[2025-08-19, 08:42:12] INFO - Done. Returned value was: None: source="airflow.task.operators.airflow.providers.standard.decorators.python._PythonDecoratedOperator"
[2025-08-19, 08:42:12] INFO - https://portal.stva.zh.ch/ecari-dispoweb/ui/app/init/#/conduite/prive/login: chan="stdout": source="task"

test_virtualenv log (fails)

::group::Log message source details: sources=["/opt/airflow/logs/dag_id=test_vault_virtualenv/run_id=scheduled__2025-08-17T00:00:00+00:00/task_id=test_virtualenv/attempt=1.log"]
[2025-08-19, 08:42:12] INFO - DAG bundles loaded: dags-folder: source="airflow.dag_processing.bundles.manager.DagBundlesManager"
[2025-08-19, 08:42:12] INFO - Filling up the DagBag from /opt/airflow/dags/test_vault_virtualenv.py: source="airflow.models.dagbag.DagBag"
[2025-08-19, 08:42:12] WARNING - /home/airflow/.local/lib/python3.12/site-packages/airflow/models/variable.py:147: DeprecationWarning: Using Variable.get from `airflow.models` is deprecated.Please use `from airflow.sdk import Variable` instead
  warnings.warn(
: source="py.warnings"
[2025-08-19, 08:42:12] INFO - Python virtual environment will be cached in /opt/airflow/venv/venv-aacc1679: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator"
[2025-08-19, 08:42:12] INFO - Executing cmd: uv venv --allow-existing --seed --python python --system-site-packages /opt/airflow/venv/venv-aacc1679: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - Output:: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - Using CPython 3.12.11 interpreter at: /usr/local/bin/python: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - Creating virtual environment with seed packages at: venv/venv-aacc1679: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO -          If the cache and target directories are on different filesystems, hardlinking may not be supported.: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO -          If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO -  + pip==25.2: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - Executing cmd: uv pip install --python /opt/airflow/venv/venv-aacc1679/bin/python -r /opt/airflow/venv/venv-aacc1679/requirements.txt: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - Output:: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - warning: Requirements file `venv/venv-aacc1679/requirements.txt` does not contain any dependencies: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - Using Python 3.12.11 environment at: venv/venv-aacc1679: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - Audited in 62ms: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - New Python virtual environment created in /opt/airflow/venv/venv-aacc1679: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator"
[2025-08-19, 08:42:12] INFO - Use 'pickle' as serializer.: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator"
[2025-08-19, 08:42:12] INFO - Executing cmd: /opt/airflow/venv/venv-aacc1679/bin/python /tmp/venv-callpa921oow/script.py /tmp/venv-callpa921oow/script.in /tmp/venv-callpa921oow/script.out /tmp/venv-callpa921oow/string_args.txt /tmp/venv-callpa921oow/termination.log /tmp/venv-callpa921oow/airflow_context.json: source="airflow.utils.process_utils"
[2025-08-19, 08:42:12] INFO - Output:: source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO - Traceback (most recent call last):: source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -   File "/tmp/venv-callpa921oow/script.py", line 54, in <module>: source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -     res = test_virtualenv(*arg_dict["args"], **arg_dict["kwargs"]): source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^: source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -   File "/tmp/venv-callpa921oow/script.py", line 26, in test_virtualenv: source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -     connection = BaseHook.get_connection("stva"): source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^: source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/hooks/base.py", line 64, in get_connection: source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -     conn = Connection.get_connection_from_secrets(conn_id): source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^: source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/connection.py", line 507, in get_connection_from_secrets: source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO -     raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined"): source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] INFO - airflow.exceptions.AirflowNotFoundException: The conn_id `stva` isn't defined: source="airflow.utils.process_utils"
[2025-08-19, 08:42:15] ERROR - Task failed with exception: source="task"
AirflowException: Process returned non-zero exit status 1.
The conn_id `stva` isn't defined

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 867 in run

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1159 in _execute_task

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 397 in wrapper

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 251 in execute

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 397 in wrapper

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 490 in execute

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 397 in wrapper

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 216 in execute

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 844 in execute_callable

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 576 in _execute_python_callable_in_subprocess

How to reproduce

Minimum reproducible DAG (requires the Vault Secrets backend to be configured, with an example "stva" connection):

from datetime import datetime

from airflow import DAG
from airflow.decorators import task
from airflow.hooks.base import BaseHook


default_args = {
    "depends_on_past": False,
    "retries": 0,
    "venv_cache_path": "/opt/airflow/venv",
}


with DAG(
    "test_vault_virtualenv",
    default_args=default_args,
    description="Test vault access for connections",
    schedule="@once",
    start_date=datetime(2025, 8, 17),
) as dag:

    @task
    def test_connection():
        connection = BaseHook.get_connection("stva")
        print(connection.host)

    @task.virtualenv(
        requirements=[],
    )
    def test_virtualenv(logical_date):
        from airflow.hooks.base import BaseHook

        connection = BaseHook.get_connection("stva")
        print(connection.host)

    test_connection()
    test_virtualenv()

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

You should use airflow.sdk.Connection.get() in Airflow 3. This is the way how you can retrieve it

[potiuk]

Converting to discussions.