Airflow 3.x: find stable combination of uvicorn version and `AIRFLOW__API__WORKERS` setting

[adwk67]

Description / Context

Airflow 3.x uses fastapi as its api-server backend, which uses uvicorn (>=0.30.0). Uvicorn switched parts of its internal architecture between 0.29 and 0.30 so that the multi-process manager uses the more resource-intensive spawning instead of forking for child processes.

See:

• Can't start airflow 3 on Ubuntu 22.04 in WSL 2 apache/airflow#50170 (comment)
• Uvicorn >=0.30 dies on Kubernetes only Kludex/uvicorn#2450
• Uvicorn >=0.30 dies on Kubernetes only Kludex/uvicorn#2450 (comment)

Problem

Using multiple workers with the default version of unicorn results in child processes being terminated as the spawn does not complete within the hard-coded timeout (5s). Possible workarounds are:

• Set AIRFLOW__API__WORKERS=1, this prevents uvicorn launching child processes avoiding the timeout issue
  • current implementation in our operator
• Downgrade to uvicorn==0.29.0, which is the last version to use forking
  • the current build references 0.34, so that is quite a downgrade

Proposal

Test that keeping the management of child processes at the api-server level (i.e. multiple api-server replicas, each with AIRFLOW__API__WORKERS=1, rather than AIRFLOW__API__WORKERS set to greater than 1 in a single api-server instance) is a feasible approach, and document in detail as such.

[lfrancke]

This is something I'd like to read up on further and potentially work with upstream.
Please hold on this until I'm back.

[lfrancke]

Do I understand it correctly that this PR Kludex/uvicorn#2397 would solve our issues because we could change the hardcoded default?

If that is the case it seems like a good case for a patch once build from source works

[lfrancke]

We discussed this internally and will do the following for now (basically as suggested by Andrew)

• We keep API__WORKERS at 1 for Airflow 3
• We add a documentation page/section explaining what to do when running into performance issues and that the correct way is to scale the webservers at the Kubernetes level
• We'll reach out to the uvicorn maintainer to see if we can get one of the two PRs merged that would give us a workaround at the "container" level add timeout_worker_is_alive as an option for main.run. Kludex/uvicorn#2397 or Add option to set worker healthcheck timeout Kludex/uvicorn#2500