docs: refresh Public Interface & align how-to guides for Airflow 3.0+ (#52297)

* docs: Update public interface documentation for Airflow 3.0+ for metadata direct access change

* docs: replace direct metadata model imports in how-to examples with airflow.sdk

* Make Public interface for Airflow 3 and add link for Airflow 2.11

* Fix PR comments

* Update airflow-core/docs/public-airflow-interface.rst

Co-Authored-By: Amogh Desai <[email protected]>

* Fix PR comments

* Remove duplicates and remove RuntimeTaskInstanceProtocol as it is not public

* Fix PR comments

---------

Co-authored-by: Amogh Desai <[email protected]>

Author: sunank200

airflow-core/docs/conf.py

@@ -120,7 +120,6 @@
 
 PACKAGES_THAT_WE_SHOULD_ADD_TO_API_DOCS = {
     "hooks",
-    "decorators",
     "example_dags",
     "executors",
     "operators",
@@ -140,15 +139,7 @@
 
 MODELS_THAT_SHOULD_BE_INCLUDED_IN_API_DOCS: set[str] = {
     "baseoperator.py",
-    "connection.py",
-    "dag.py",
-    "dagrun.py",
-    "dagbag.py",
     "param.py",
-    "taskinstance.py",
-    "taskinstancekey.py",
-    "variable.py",
-    "xcom.py",
 }
 
 

airflow-core/docs/core-concepts/params.rst

@@ -32,7 +32,7 @@ If the user-supplied values don't pass validation, Airflow shows a warning inste
 DAG-level Params
 ----------------
 
-To add Params to a :class:`~airflow.models.dag.DAG`, initialize it with the ``params`` kwarg.
+To add Params to a :class:`~airflow.sdk.DAG`, initialize it with the ``params`` kwarg.
 Use a dictionary that maps Param names to either a :class:`~airflow.sdk.definitions.param.Param` or an object indicating the parameter's default value.
 
 .. code-block::

airflow-core/docs/core-concepts/variables.rst

@@ -33,6 +33,20 @@ To use them, just import and call ``get`` on the Variable model::
     # Returns the value of default (None) if the variable is not set
     baz = Variable.get("baz", default=None)
 
+You can also access variables through the Task Context using
+:func:`~airflow.sdk.get_current_context`:
+
+.. code-block:: python
+
+    from airflow.sdk import get_current_context
+
+
+    def my_task():
+        context = get_current_context()
+        var = context["var"]
+        my_variable = var.get("my_variable_name")
+        return my_variable
+
 You can also use them from :ref:`templates <concepts:jinja-templating>`::
 
     # Raw value

airflow-core/docs/core-concepts/xcoms.rst

@@ -25,6 +25,9 @@ XComs (short for "cross-communications") are a mechanism that let :doc:`tasks` t
 
 An XCom is identified by a ``key`` (essentially its name), as well as the ``task_id`` and ``dag_id`` it came from. They can have any serializable value (including objects that are decorated with ``@dataclass`` or ``@attr.define``, see :ref:`TaskFlow arguments <concepts:arbitrary-arguments>`:), but they are only designed for small amounts of data; do not use them to pass around large values, like dataframes.
 
+XCom operations should be performed through the Task Context using
+:func:`~airflow.sdk.get_current_context`. Directly updating using XCom database model is not possible.
+
 XComs are explicitly "pushed" and "pulled" to/from their storage using the ``xcom_push`` and ``xcom_pull`` methods on Task Instances.
 
 To push a value within a task called **"task-1"** that will be used by another task:
@@ -73,8 +76,6 @@ An example of pushing multiple XComs and pulling them individually:
         # Pulling entire xcom data from push_multiple task
         data = context["ti"].xcom_pull(task_ids="push_multiple", key="return_value")
 
-
-
 .. note::
 
   If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
@@ -91,7 +92,7 @@ Custom XCom Backends
 
 The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
 
-If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
+If you want to implement your own backend, you should subclass :class:`~airflow.sdk.bases.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
 
 You can override the ``purge`` method in the ``BaseXCom`` class to have control over purging the xcom data from the custom backend. This will be called as part of ``delete``.
 
@@ -104,6 +105,6 @@ If you can exec into a terminal in an Airflow container, you can then print out
 
 .. code-block:: python
 
-    from airflow.models.xcom import XCom
+    from airflow.sdk.execution_time.xcom import XCom
 
     print(XCom.__name__)

airflow-core/docs/howto/connection.rst

@@ -22,7 +22,7 @@ Managing Connections
 
   For an overview of hooks and connections, see :doc:`/authoring-and-scheduling/connections`.
 
-Airflow's :class:`~airflow.models.connection.Connection` object is used for storing credentials and other information necessary for connecting to external services.
+Airflow's :class:`~airflow.sdk.Connection` object is used for storing credentials and other information necessary for connecting to external services.
 
 Connections may be defined in the following ways:
 
@@ -77,7 +77,7 @@ convenience property :py:meth:`~airflow.models.connection.Connection.as_json`. I
 
 .. code-block:: pycon
 
-    >>> from airflow.models.connection import Connection
+    >>> from airflow.sdk import Connection
     >>> c = Connection(
     ...     conn_id="some_conn",
     ...     conn_type="mysql",
@@ -94,7 +94,7 @@ In addition, same approach could be used to convert Connection from URI format t
 
 .. code-block:: pycon
 
-    >>> from airflow.models.connection import Connection
+    >>> from airflow.sdk import Connection
     >>> c = Connection(
     ...     conn_id="awesome_conn",
     ...     description="Example Connection",

airflow-core/docs/howto/custom-operator.rst

@@ -24,7 +24,7 @@ Creating a custom Operator
 Airflow allows you to create new operators to suit the requirements of you or your team.
 This extensibility is one of the many features which make Apache Airflow powerful.
 
-You can create any operator you want by extending the :class:`airflow.models.baseoperator.BaseOperator`
+You can create any operator you want by extending the public SDK base class :class:`~airflow.sdk.BaseOperator`.
 
 There are two methods that you need to override in a derived class: