Skip to content

[airflow] Expand module path check to individual symbols (AIR302) #17278

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Apr 8, 2025

Conversation

Lee-W
Copy link
Contributor

@Lee-W Lee-W commented Apr 7, 2025

Summary

Improvement

Expand the following moved module into individual symbols.

  • airflow.triggers.temporal
  • airflow.triggers.file
  • airflow.triggers.external_task
  • airflow.hooks.subprocess
  • airflow.hooks.package_index
  • airflow.hooks.filesystem
  • airflow.sensors.weekday
  • airflow.sensors.time_delta
  • airflow.sensors.time_sensor
  • airflow.sensors.date_time
  • airflow.operators.weekday
  • airflow.operators.datetime
  • airflow.operators.bash

This removes Replacement::ImportPathMoved.

Fix

During the expansion, the following paths were also fixed

  • airflow.sensors.s3_key_sensor.S3KeySensor → airflow.providers.amazon.aws.sensors.S3KeySensor
  • airflow.operators.sql.SQLThresholdCheckOperator → airflow.providers.common.sql.operators.sql.SQLThresholdCheckOperator
  • airflow.hooks.druid_hook.DruidDbApiHook → airflow.providers.apache.druid.hooks.druid.DruidDbApiHook
  • airflow.hooks.druid_hook.DruidHook → airflow.providers.apache.druid.hooks.druid.DruidHook
  • airflow.kubernetes.pod_generator.extend_object_field → airflow.providers.cncf.kubernetes.pod_generator.extend_object_field
  • airflow.kubernetes.pod_launcher.PodLauncher → airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodLauncher
  • airflow.kubernetes.pod_launcher.PodStatus → airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodStatus
  • airflow.kubernetes.pod_generator.PodDefaults → airflow.providers.cncf.kubernetes.pod_generator.PodDefaults
  • airflow.kubernetes.pod_launcher_deprecated.PodDefaults → airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodDefaults

Refactor

As many symbols are moved into the same module, SourceModuleMovedToProvider is introduced for grouping similar logic

Test Plan

Copy link
Contributor

github-actions bot commented Apr 7, 2025

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

ℹ️ ecosystem check detected linter changes. (+2 -2 violations, +0 -0 fixes in 1 projects; 54 projects unchanged)

apache/airflow (+2 -2 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --no-fix --output-format concise --preview --select ALL

- performance/src/performance_dags/performance_dag/performance_dag.py:95:13: AIR302 Import path `airflow.operators.bash` is moved into `standard` provider in Airflow 3.0;
+ performance/src/performance_dags/performance_dag/performance_dag.py:95:13: AIR302 `airflow.operators.bash.BashOperator` is moved into `standard` provider in Airflow 3.0;
- providers/edge/src/airflow/providers/edge/example_dags/integration_test.py:116:24: AIR302 Import path `airflow.operators.bash` is moved into `standard` provider in Airflow 3.0;
+ providers/edge/src/airflow/providers/edge/example_dags/integration_test.py:116:24: AIR302 `airflow.operators.bash.BashOperator` is moved into `standard` provider in Airflow 3.0;

Changes by rule (1 rules affected)

code total + violation - violation + fix - fix
AIR302 4 2 2 0 0

@Lee-W Lee-W marked this pull request as draft April 7, 2025 15:08
@Lee-W Lee-W force-pushed the get-rid-of-ImportPathMoved branch from dff2a88 to 08a8742 Compare April 7, 2025 15:10
@Lee-W Lee-W changed the title Get rid of import path moved [airflow] Expand module path check to individual symbols (AIR302) Apr 7, 2025
@Lee-W Lee-W marked this pull request as ready for review April 7, 2025 15:12
@Lee-W
Copy link
Contributor Author

Lee-W commented Apr 7, 2025

@ntBre Hey, it would be nice if we could get this one reviewed. 🙏 The amount of change is enormous, but I tried to split the PR into 20 separate commits, so I hope it will be easier to review that way

@ntBre ntBre added rule Implementing or modifying a lint rule preview Related to preview mode features labels Apr 7, 2025
Copy link
Contributor

@ntBre ntBre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! This was easy to review commit-by-commit. I had one suggestion, but I'm happy enough with the current version if you are.

Lee-W added 17 commits April 8, 2025 18:03
* airflow.sensors.s3_key_sensor.S3KeySensor → airflow.providers.amazon.aws.sensors.S3KeySensor
* airflow.operators.sql.SQLThresholdCheckOperator → airflow.providers.common.sql.operators.sql.SQLThresholdCheckOperator
* airflow.hooks.druid_hook.DruidDbApiHook → airflow.providers.apache.druid.hooks.druid.DruidDbApiHook
* airflow.hooks.druid_hook.DruidHook → airflow.providers.apache.druid.hooks.druid.DruidHook
* airflow.kubernetes.pod_generator.extend_object_field → airflow.providers.cncf.kubernetes.pod_generator.extend_object_field
* airflow.kubernetes.pod_launcher.PodLauncher → airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodLauncher
* airflow.kubernetes.pod_launcher.PodStatus → airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodStatus
* airflow.kubernetes.pod_generator.PodDefaults → airflow.providers.cncf.kubernetes.pod_generator.PodDefaults
* airflow.kubernetes.pod_launcher_deprecated.PodDefaults → airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodDefaults
@Lee-W Lee-W force-pushed the get-rid-of-ImportPathMoved branch from b1d9dae to 2b039f0 Compare April 8, 2025 10:03
@ntBre ntBre changed the title [airflow] Expand module path check to individual symbols (AIR302) [airflow] Expand module path check to individual symbols (AIR302) Apr 8, 2025
@ntBre ntBre merged commit dc02732 into astral-sh:main Apr 8, 2025
22 checks passed
dcreager added a commit that referenced this pull request Apr 9, 2025
* origin/main:
  [red-knot] Default `python-platform` to current platform (#17183)
  [red-knot] Add new 'unreachable code' test case (#17306)
  [red-knot] mypy_primer: Run on `async-utils` (#17303)
  [red-knot] Add custom `__setattr__` support (#16748)
  [red-knot] Add `__init__` arguments check when doing `try_call` on a class literal (#16512)
  [`flake8-pie`] Avoid false positive for multiple assignment with `auto()` (`PIE796`) (#17274)
  [syntax-errors] Async comprehension in sync comprehension (#17177)
  [`airflow`] Expand module path check to individual symbols (`AIR302`) (#17278)
  [syntax-errors] Check annotations in annotated assignments (#17283)
  [syntax-errors] Extend annotation checks to `await` (#17282)
  [red-knot] Add support for `assert_never` (#17287)
  [`flake8-pytest-style`] Avoid false positive for legacy form of `pytest.raises` (`PT011`) (#17231)
  [red-knot] Do not show types for literal expressions on hover (#17290)
  [red-knot] Fix dead-code clippy warning (#17291)
  [red-knot] Reachability analysis (#17199)
  [red-knot] Don't use latency-sensitive for handlers (#17227)
dcreager added a commit that referenced this pull request Apr 9, 2025
* dcreager/special-class: (26 commits)
  lint
  Add TODO about property test data
  Better todos
  Narrow type(generic) better
  More Python-like displays for specializations
  Add xfail for generic method inside generic class
  Comment other non-specializations
  Explain self_instance not being specialized
  Generic aliases are literals in type display
  Better TODO fallback type
  [red-knot] Default `python-platform` to current platform (#17183)
  [red-knot] Add new 'unreachable code' test case (#17306)
  [red-knot] mypy_primer: Run on `async-utils` (#17303)
  [red-knot] Add custom `__setattr__` support (#16748)
  [red-knot] Add `__init__` arguments check when doing `try_call` on a class literal (#16512)
  [`flake8-pie`] Avoid false positive for multiple assignment with `auto()` (`PIE796`) (#17274)
  [syntax-errors] Async comprehension in sync comprehension (#17177)
  [`airflow`] Expand module path check to individual symbols (`AIR302`) (#17278)
  [syntax-errors] Check annotations in annotated assignments (#17283)
  [syntax-errors] Extend annotation checks to `await` (#17282)
  ...
Glyphack pushed a commit to Glyphack/ruff that referenced this pull request Apr 9, 2025
…astral-sh#17278)

## Summary

### Improvement
Expand the following moved module into individual symbols.

* airflow.triggers.temporal
* airflow.triggers.file
* airflow.triggers.external_task
* airflow.hooks.subprocess
* airflow.hooks.package_index
* airflow.hooks.filesystem
* airflow.sensors.weekday
* airflow.sensors.time_delta
* airflow.sensors.time_sensor
* airflow.sensors.date_time
* airflow.operators.weekday
* airflow.operators.datetime
* airflow.operators.bash 

This removes `Replacement::ImportPathMoved`.

## Fix
During the expansion, the following paths were also fixed

* airflow.sensors.s3_key_sensor.S3KeySensor →
airflow.providers.amazon.aws.sensors.S3KeySensor
* airflow.operators.sql.SQLThresholdCheckOperator →
airflow.providers.common.sql.operators.sql.SQLThresholdCheckOperator
* airflow.hooks.druid_hook.DruidDbApiHook →
airflow.providers.apache.druid.hooks.druid.DruidDbApiHook
* airflow.hooks.druid_hook.DruidHook →
airflow.providers.apache.druid.hooks.druid.DruidHook
* airflow.kubernetes.pod_generator.extend_object_field →
airflow.providers.cncf.kubernetes.pod_generator.extend_object_field
* airflow.kubernetes.pod_launcher.PodLauncher →
airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodLauncher
* airflow.kubernetes.pod_launcher.PodStatus →
airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodStatus
* airflow.kubernetes.pod_generator.PodDefaults →
airflow.providers.cncf.kubernetes.pod_generator.PodDefaults
* airflow.kubernetes.pod_launcher_deprecated.PodDefaults →
airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodDefaults

### Refactor
As many symbols are moved into the same module,
`SourceModuleMovedToProvider` is introduced for grouping similar logic

## Test Plan
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preview Related to preview mode features rule Implementing or modifying a lint rule
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants