Add basic dag-bundles docs; webserver->api server in security model (#48600)

Author: jedcunningham

airflow-core/docs/administration-and-deployment/dag-bundles.rst

@@ -0,0 +1,129 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+Dag Bundles
+===========
+
+Dag bundle are a collection of dags and other files (think the Airflow 2 dags folder). Unlike Airflow 2, where dags were required to be on local disk and getting
+the dags there was the sole responsibility of the deployment manager, Airflow 3 is now able to pull dags from external systems
+as well. And since dag bundles support versioning, it also allows Airflow to run a task using a specific version
+of the dag bundle, allowing for a dag run to use the same code for the whole run, even if the dag is updated mid way through the run.
+
+What's in a dag bundle? One or more dag files along with their associated files, such as
+other Python scripts, configuration files, or other resources. By keeping the bundle at a higher level, it allows for versioning
+everything the dag needs to run.
+
+Dag bundles can source the dags from various locations, such as local directories, Git repositories, or other external systems.
+Deployment administrators can also write their own dag bundle classes to support custom sources.
+You can also define more than 1 dag bundle in an Airflow deployments, allowing for better organization of your dags.
+
+Why Are dag bundles important?
+------------------------------
+
+- **Version Control**: By supporting versioning, dag bundles allow dag runs to use the same code for the whole run, even if the dag is updated mid way through the run.
+- **Scalability**: With dag bundles, Airflow can efficiently manage large numbers of DAGs by organizing them into logical units.
+- **Flexibility**: Dag bundles enable seamless integration with external systems, such as Git repositories, to source dags.
+
+Types of dag bundles
+--------------------
+Airflow supports multiple types of dag Bundles, each catering to specific use cases:
+
+**airflow.dag_processing.bundles.local.LocalDagBundle**
+    These bundles reference a local directory containing DAG files. They are ideal for development and testing environments, but do not support versioning of the bundle, meaning tasks always run using the latest code.
+
+**airflow.providers.git.bundles.git.GitDagBundle**
+    These bundles integrate with Git repositories, allowing Airflow to fetch dags directly from a repository.
+
+Configuring dag bundles
+-----------------------
+
+Dag bundles are configured in :ref:`config:dag_processor__dag_bundle_config_list`. You can add one or more dag bundles here.
+
+By default, Airflow adds a local dag bundle, which is the same as the old dags folder. This is done for backwards compatibility, and you can remove it if you do not want to use it. You can also keep it and add other dag bundles, such as a git dag bundle.
+
+For example, adding multiple dag bundles to your ``airflow.cfg`` file:
+
+.. code-block:: ini
+
+    [dag_processor]
+    dag_bundle_config_list = [
+            {
+                "name": "my_git_repo",
+                "classpath": "airflow.dag_processing.bundles.git.GitDagBundle",
+                "kwargs": {"tracking_ref": "main", "git_conn_id": "my_git_conn"}
+            }
+            {
+              "name": "dags-folder",
+              "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle",
+              "kwargs": {}
+            }
+        ]
+
+.. note::
+
+    The whitespace, particularly on the last line, is important so a multi-line value works properly. More details can be found in the
+    the `configparser docs <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure>`_.
+
+You can also override the :ref:`config:dag_processor__refresh_interval` per dag bundle by passing it in kwargs.
+This controls how often the dag processor refreshes, or looks for new files, in the dag bundles.
+
+Writing custom dag bundles
+--------------------------
+
+When implementing your own dag bundle by extending the ``BaseDagBundle`` class, there are several methods you must implement. Below is a guide to help you implement a custom dag bundle.
+
+Abstract Methods
+~~~~~~~~~~~~~~~~
+The following methods are abstract and must be implemented in your custom bundle class:
+
+**path**
+    This property should return a ``Path`` to the directory where the dag files for this bundle are stored.
+    Airflow uses this property to locate the DAG files for processing.
+
+**get_current_version**
+    This method should return the current version of the bundle as a string.
+    Airflow will use pass this version to ``__init__`` later to get this version of the bundle again when it runs tasks.
+    If versioning is not supported, it should return ``None``.
+
+**refresh**
+    This method should handle refreshing the bundle's contents from its source (e.g., pulling the latest changes from a remote repository).
+    This is used by the dag processor periodically to ensure that the bundle is up-to-date.
+
+Optional Methods
+~~~~~~~~~~~~~~~~
+In addition to the abstract methods, you may choose to override the following methods to customize the behavior of your bundle:
+
+**__init__**
+    This method can be extended to initialize the bundle with extra parameters, such as ``tracking_ref`` for the ``GitDagBundle``.
+    It should also call the parent class's ``__init__`` method to ensure proper initialization.
+    Expensive operations, such as network calls, should be avoided in this method to prevent delays during the bundle's instantiation, and done
+    in the ``initialize`` method instead.
+
+**initialize**
+    This method is called before the bundle is first used in the dag processor or worker. It allows you to perform expensive operations only when the bundle's content is accessed.
+
+**view_url**
+    This method should return a URL as a string to view the bundle on an external system (e.g., a Git repository's web interface).
+
+Other Considerations
+~~~~~~~~~~~~~~~~~~~~
+
+- **Versioning**: If your bundle supports versioning, ensure that ``initialize``, ``get_current_version`` and ``refresh`` are implemented to handle version-specific logic.
+
+- **Concurrency**: Workers may create many bundles simultaneously, and does nothing to serialize calls to the bundle objects. Thus, the bundle class must handle locking if
+  that is problematic for the underlying technology. For example, if you are cloning a git repo, the bundle class is responsible for locking to ensure only 1 bundle
+  object is cloning at a time. There is a ``lock`` method in the base class that can be used for this purpose, if necessary.

airflow-core/docs/administration-and-deployment/index.rst

@@ -28,6 +28,7 @@ This section contains information about deploying dags into production and the a
     kubernetes
     lineage
     listeners
+    dag-bundles
     dag-serialization
     modules_management
     scheduler

airflow-core/docs/core-concepts/dags.rst

@@ -147,7 +147,7 @@ Chain can also do *pairwise* dependencies for lists the same size (this is diffe
 Loading dags
 ------------
 
-Airflow loads dags from Python source files, which it looks for inside its configured ``DAG_FOLDER``. It will take each file, execute it, and then load any DAG objects from that file.
+Airflow loads dags from Python source files in dag bundles. It will take each file, execute it, and then load any DAG objects from that file.
 
 This means you can define multiple dags per Python file, or even spread one very complex DAG across multiple Python files using imports.
 
@@ -164,11 +164,11 @@ While both DAG constructors get called when the file is accessed, only ``dag_1``
 
 .. note::
 
-    When searching for dags inside the ``DAG_FOLDER``, Airflow only considers Python files that contain the strings ``airflow`` and ``dag`` (case-insensitively) as an optimization.
+    When searching for dags inside the dag bundle, Airflow only considers Python files that contain the strings ``airflow`` and ``dag`` (case-insensitively) as an optimization.
 
     To consider all Python files instead, disable the ``DAG_DISCOVERY_SAFE_MODE`` configuration flag.
 
-You can also provide an ``.airflowignore`` file inside your ``DAG_FOLDER``, or any of its subfolders, which describes patterns of files for the loader to ignore. It covers the directory it's in plus all subfolders underneath it. See  :ref:`.airflowignore <concepts:airflowignore>` below for details of the file syntax.
+You can also provide an ``.airflowignore`` file inside your dag bundle, or any of its subfolders, which describes patterns of files for the loader to ignore. It covers the directory it's in plus all subfolders underneath it. See  :ref:`.airflowignore <concepts:airflowignore>` below for details of the file syntax.
 
 In the case where the ``.airflowignore`` does not meet your needs and you want a more flexible way to control if a python file needs to be parsed by airflow, you can plug your callable by setting ``might_contain_dag_callable`` in the config file.
 Note, this callable will replace the default Airflow heuristic, i.e. checking if the strings ``airflow`` and ``dag`` (case-insensitively) are present in the python file.
@@ -691,7 +691,7 @@ Packaging dags
 
 While simpler dags are usually only in a single Python file, it is not uncommon that more complex dags might be spread across multiple files and have dependencies that should be shipped with them ("vendored").
 
-You can either do this all inside of the ``DAG_FOLDER``, with a standard filesystem layout, or you can package the DAG and all of its Python files up as a single zip file. For instance, you could ship two dags along with a dependency they need as a zip file with the following contents::
+You can either do this all inside of the dag bundle, with a standard filesystem layout, or you can package the DAG and all of its Python files up as a single zip file. For instance, you could ship two dags along with a dependency they need as a zip file with the following contents::
 
     my_dag1.py
     my_dag2.py
@@ -711,7 +711,7 @@ In general, if you have a complex set of compiled dependencies and modules, you
 ``.airflowignore``
 ------------------
 
-An ``.airflowignore`` file specifies the directories or files in ``DAG_FOLDER``
+An ``.airflowignore`` file specifies the directories or files in the dag bundle
 or ``PLUGINS_FOLDER`` that Airflow should intentionally ignore. Airflow supports
 two syntax flavors for patterns in the file, as specified by the ``DAG_IGNORE_FILE_SYNTAX``
 configuration parameter (*added in Airflow 2.3*): ``regexp`` and ``glob``.
@@ -740,7 +740,7 @@ match any of the patterns would be ignored (under the hood, ``Pattern.search()``
 to match the pattern). Use the ``#`` character to indicate a comment; all characters
 on lines starting with ``#`` will be ignored.
 
-The ``.airflowignore`` file should be put in your ``DAG_FOLDER``. For example, you can prepare
+The ``.airflowignore`` file should be put in your dag bundle. For example, you can prepare
 a ``.airflowignore`` file with the ``glob`` syntax
 
 .. code-block::
@@ -749,12 +749,12 @@ a ``.airflowignore`` file with the ``glob`` syntax
     tenant_[0-9]*
 
 Then files like ``project_a_dag_1.py``, ``TESTING_project_a.py``, ``tenant_1.py``,
-``project_a/dag_1.py``, and ``tenant_1/dag_1.py`` in your ``DAG_FOLDER`` would be ignored
+``project_a/dag_1.py``, and ``tenant_1/dag_1.py`` in your dag bundle would be ignored
 (If a directory's name matches any of the patterns, this directory and all its subfolders
 would not be scanned by Airflow at all. This improves efficiency of DAG finding).
 
 The scope of a ``.airflowignore`` file is the directory it is in plus all its subfolders.
-You can also prepare ``.airflowignore`` file for a subfolder in ``DAG_FOLDER`` and it
+You can also prepare ``.airflowignore`` file for a subfolder in your dag bundle and it
 would only be applicable for that subfolder.
 
 DAG Dependencies