Skip to content

Commit 80168f6

Browse files
ethan-oroHelper Bot
and
Helper Bot
committed
[airflow] upgrade to composer-2.7.1-airflow-2.7.3, python 3.11 (Recidiviz/recidiviz-data#29712)
## Description of the change Recidiviz/recidiviz-data#27852 upgraded to `composer-2.6.2-airflow-2.6.3` which failed during the tf step and was reverted in Recidiviz/recidiviz-data#29696 The commits in this PR are as follows -- 1. un-reverts Recidiviz/recidiviz-data#27852 2. upgrades to `composer-2.7.1-airflow-2.7.3`, copying requirements over from [version page](https://cloud.google.com/composer/docs/concepts/versioning/composer-versions#images) 3. airflow pipenv lock workflow 4. removes a few imports and checks since [Recidiviz/recidiviz-data#32731](apache/airflow#32731) opened by our very own @ohaibbq is included in 2.7.1 read through the [release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-2-7-0-2023-08-18), nothing in our usage seems to be deprecated; there are a few things that might be of interest to folks: - _Create metrics to track Scheduled->Queued->Running task state transition times ([Recidiviz/recidiviz-data#30612](apache/airflow#30612 will be nice for us to be able to see this re: k8s spin up time - _Add OpenTelemetry to Airflow ([AIP-49](https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-49+milestone%3A%22Airflow+2.7.0%22))_ - _The trigger UI form is skipped in web UI if no parameters are defined in a DAG ([Recidiviz/recidiviz-data#33351](apache/airflow#33351: not sure if folks run ad-hoc dags using the UI instead of `trigger_state_specific_calculation/ingest_dag` but we need to add `show_trigger_form_if_no_params` if we want to still be able to trigger via the UI would love a second set of eyes around these warning logs that occur after tests run, see [here](https://github.com/Recidiviz/recidiviz-data/actions/runs/9071731702/job/24926024866?pr=29712#step:6:53). also occurred locally but i couldn't isolate the cause of the issue ``` Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/logging/__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: I/O operation on closed file. Call stack: File "/home/runner/.local/share/virtualenvs/airflow-jCIRjM3K/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 509, in <lambda> and _finalize_fairy( File "/home/runner/.local/share/virtualenvs/airflow-jCIRjM3K/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 791, in _finalize_fairy pool.logger.error( Message: 'Exception during reset or similar' Arguments: () --- Logging error --- Traceback (most recent call last): File "/home/runner/.local/share/virtualenvs/airflow-jCIRjM3K/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy fairy._reset(pool, transaction_was_reset) File "/home/runner/.local/share/virtualenvs/airflow-jCIRjM3K/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset pool._dialect.do_rollback(self) File "/home/runner/.local/share/virtualenvs/airflow-jCIRjM3K/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback dbapi_connection.rollback() psycopg2.OperationalError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/logging/__init__.py", line 1113, in emit stream.write(msg + self.terminator) ValueError: I/O operation on closed file. Call stack: File "/home/runner/.local/share/virtualenvs/airflow-jCIRjM3K/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 509, in <lambda> and _finalize_fairy( File "/home/runner/.local/share/virtualenvs/airflow-jCIRjM3K/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 791, in _finalize_fairy pool.logger.error( Message: 'Exception during reset or similar' Arguments: () ``` ## Type of change > All pull requests must have at least one of the following labels applied (otherwise the PR will fail): | Label | Description | |----------------------------- |----------------------------------------------------------------------------------------------------------- | | Type: Bug | non-breaking change that fixes an issue | | Type: Feature | non-breaking change that adds functionality | | Type: Breaking Change | fix or feature that would cause existing functionality to not work as expected | | Type: Non-breaking refactor | change addresses some tech debt item or prepares for a later change, but does not change functionality | | Type: Configuration Change | adjusts configuration to achieve some end related to functionality, development, performance, or security | | Type: Dependency Upgrade | upgrades a project dependency - these changes are not included in release notes | ## Related issues Closes Recidiviz/recidiviz-data#29554 ## Checklists ### Development **This box MUST be checked by the submitter prior to merging**: - [x] **Double- and triple-checked that there is no Personally Identifiable Information (PII) being mistakenly added in this pull request** These boxes should be checked by the submitter prior to merging: - [ ] Tests have been written to cover the code changed/added as part of this pull request ### Code review These boxes should be checked by reviewers prior to merging: - [x] This pull request has a descriptive title and information useful to a reviewer - [x] Potential security implications or infrastructural changes have been considered, if relevant --------- Co-authored-by: Helper Bot <[email protected]> GitOrigin-RevId: 355d050265f490df1779ff773533f5aefd41de6b
1 parent 9170adb commit 80168f6

File tree

6 files changed

+3668
-3062
lines changed

6 files changed

+3668
-3062
lines changed

.github/actions/airflow-python-setup/action.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ runs:
44
steps:
55
- uses: actions/setup-python@v2
66
with:
7-
python-version: "3.8.12"
7+
python-version: "3.11.8"
88
- name: Install pipenv
99
run: pip install pipenv
1010
shell: bash
@@ -13,13 +13,13 @@ runs:
1313
id: pipenv-caching
1414
with:
1515
path: ~/.local/share/virtualenvs/
16-
key: ${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-pipenv-airflow-v3-${{ hashFiles('**/recidiviz/airflow/Pipfile.lock') }}
16+
key: ${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-pipenv-airflow-v5-${{ hashFiles('**/recidiviz/airflow/Pipfile.lock') }}
1717
# If we haven't cached a pipenv for the new version of the lock file this will
1818
# restore the cache for the most recent version. Since most of the time our
1919
# lock file updates only change a small number of packages, this speeds up
2020
# the sync step significantly.
2121
restore-keys: |
22-
${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-pipenv-airflow-v3-
22+
${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-pipenv-airflow-v5-
2323
- name: Sync pipenv
2424
if: steps.pipenv-caching.outputs.cache-hit != 'true'
2525
# If we didn't find an exact pipenv match, we have to `sync.` Because we may have

0 commit comments

Comments
 (0)