Memory profiling for Airflow with Memray.
This section contains the list of all the available Airflow Memray configurations that you can set in airflow.cfg file or using environment variables.
The base folder under which Airflow Memray will store profiling results. Possible values can be anything what is supported by Airflow Object Storage.
If it refers to a local file system path, then it must be accessible by the task and the webserver.
Default: "file:///tmp/airflow/memray"
Environment Variable: AIRFLOW__MEMRAY__BASE_FOLDER
The Airflow Connection to use if base_folder is set to a remote cloud storage location.
Default: None
Environment Variable: AIRFLOW__MEMRAY__STORAGE_CONN_ID
The tasks to be profiled as a comma separated list of wildcard pattern as implemented by the fnmatch module. The pattern are applied against the full task ID in the form <dag_id>.<task_id>.
Set it to "*" to profile all tasks.
Default: ""
Environment Variable: AIRFLOW__MEMRAY__TASKS
I have given a presentation about this package at Airflow Summit 2024.
You can visit the official page of the presentation or directly watch video on YouTube by clicking on the following picture:
