Skip to content

[ty] Support declaration-only attributes #19048

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 7, 2025

Conversation

iyakushev
Copy link
Contributor

@iyakushev iyakushev commented Jun 30, 2025

Summary

Following ty issue #698 this PR adds support for declarations.
I'd say the implementation is rather naive, so I would use it as a starting point for a gradual improvement. I tried to check declarations before bindings and it seemed to resolve the issue.

closes #698

Test Plan

Tested against mdtest (specifically attributes).

@ntBre ntBre added the ty Multi-file analysis & type inference label Jun 30, 2025
Copy link
Contributor

github-actions bot commented Jun 30, 2025

mypy_primer results

Changes were detected when running on open source projects
trio (https://github.com/python-trio/trio)
- error[unresolved-attribute] src/trio/_tests/test_ssl.py:106:8: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] src/trio/_tests/test_ssl.py:108:10: Type `FixtureRequest` has no attribute `param`
- Found 801 diagnostics
+ Found 799 diagnostics

kornia (https://github.com/kornia/kornia)
- error[unresolved-attribute] kornia/nerf/samplers.py:230:13: Unresolved attribute `_x` on type `Points2D_FlatTensors`.
- error[unresolved-attribute] kornia/nerf/samplers.py:231:13: Unresolved attribute `_y` on type `Points2D_FlatTensors`.
- error[unresolved-attribute] kornia/nerf/samplers.py:233:13: Unresolved attribute `_x` on type `Points2D_FlatTensors`.
- error[unresolved-attribute] kornia/nerf/samplers.py:233:57: Type `Points2D_FlatTensors` has no attribute `_x`
- error[unresolved-attribute] kornia/nerf/samplers.py:234:13: Unresolved attribute `_y` on type `Points2D_FlatTensors`.
- error[unresolved-attribute] kornia/nerf/samplers.py:234:57: Type `Points2D_FlatTensors` has no attribute `_y`
- error[unresolved-attribute] kornia/nerf/samplers.py:259:30: Type `Points2D_FlatTensors` has no attribute `_x`
- error[unresolved-attribute] kornia/nerf/samplers.py:259:58: Type `Points2D_FlatTensors` has no attribute `_y`
- Found 800 diagnostics
+ Found 792 diagnostics

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- TOTAL MEMORY USAGE: ~88MB
+ TOTAL MEMORY USAGE: ~97MB

yarl (https://github.com/aio-libs/yarl)
- error[unresolved-attribute] tests/test_quoting.py:18:16: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] tests/test_quoting.py:22:16: Type `FixtureRequest` has no attribute `param`
+ warning[unused-ignore-comment] tests/test_quoting.py:32:31: Unused blanket `type: ignore` directive
+ warning[unused-ignore-comment] tests/test_quoting.py:36:31: Unused blanket `type: ignore` directive

flake8 (https://github.com/pycqa/flake8)
-     memo fields = ~66MB
+     memo fields = ~60MB

pylox (https://github.com/sco1/pylox)
-     memo fields = ~54MB
+     memo fields = ~49MB

websockets (https://github.com/aaugustin/websockets)
- error[unresolved-attribute] src/websockets/asyncio/router.py:89:13: Unresolved attribute `handler` on type `ServerConnection`.
- error[unresolved-attribute] src/websockets/asyncio/router.py:89:33: Unresolved attribute `handler_kwargs` on type `ServerConnection`.
- error[unresolved-attribute] src/websockets/asyncio/router.py:94:26: Type `ServerConnection` has no attribute `handler`
- error[unresolved-attribute] src/websockets/asyncio/router.py:94:59: Type `ServerConnection` has no attribute `handler_kwargs`
- error[unresolved-attribute] src/websockets/asyncio/server.py:980:9: Unresolved attribute `username` on type `ServerConnection`.
- error[unresolved-attribute] src/websockets/sync/router.py:89:13: Unresolved attribute `handler` on type `ServerConnection`.
- error[unresolved-attribute] src/websockets/sync/router.py:89:33: Unresolved attribute `handler_kwargs` on type `ServerConnection`.
- error[unresolved-attribute] src/websockets/sync/router.py:94:20: Type `ServerConnection` has no attribute `handler`
- error[unresolved-attribute] src/websockets/sync/router.py:94:53: Type `ServerConnection` has no attribute `handler_kwargs`
- error[unresolved-attribute] src/websockets/sync/server.py:762:9: Unresolved attribute `username` on type `ServerConnection`.
- Found 65 diagnostics
+ Found 55 diagnostics

poetry (https://github.com/python-poetry/poetry)
- error[unresolved-attribute] tests/console/commands/test_init.py:316:31: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] tests/installation/test_installer.py:117:48: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] tests/installation/test_installer.py:118:11: Type `FixtureRequest` has no attribute `param`
- Found 939 diagnostics
+ Found 936 diagnostics

schemathesis (https://github.com/schemathesis/schemathesis)
-     memo fields = ~129MB
+     memo fields = ~142MB

typeshed-stats (https://github.com/AlexWaygood/typeshed-stats)
- error[unresolved-attribute] tests/conftest.py:283:12: Type `FixtureRequest` has no attribute `param`
+ warning[unused-ignore-comment] tests/test___all__.py:26:27: Unused blanket `type: ignore` directive
- error[unresolved-attribute] tests/test__cli.py:204:20: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] tests/test_gather.py:174:24: Type `FixtureRequest` has no attribute `param`
- Found 26 diagnostics
+ Found 24 diagnostics

urllib3 (https://github.com/urllib3/urllib3)
- error[unresolved-attribute] src/urllib3/connectionpool.py:553:13: Type `BaseHTTPResponse` has no attribute `length_remaining`
- error[unresolved-attribute] test/conftest.py:289:28: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] test/conftest.py:373:8: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] test/conftest.py:383:15: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] test/conftest.py:385:12: Type `FixtureRequest` has no attribute `param`
- Found 438 diagnostics
+ Found 433 diagnostics

scrapy (https://github.com/scrapy/scrapy)
-     memo fields = ~189MB
+     memo fields = ~207MB

psycopg (https://github.com/psycopg/psycopg)
- error[unresolved-attribute] psycopg_pool/psycopg_pool/base.py:201:9: Unresolved attribute `_expire_at` on type `BaseConnection[Any]`.
- Found 562 diagnostics
+ Found 561 diagnostics

pytest (https://github.com/pytest-dev/pytest)
- error[unresolved-attribute] src/_pytest/python.py:1110:12: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] testing/test_pathlib.py:120:17: Type `FixtureRequest` has no attribute `param`
- Found 527 diagnostics
+ Found 525 diagnostics

altair (https://github.com/vega/altair)
-     memo fields = ~228MB
+     memo fields = ~251MB

rotki (https://github.com/rotki/rotki)
- error[no-matching-overload] rotkehlchen/tests/api/test_eth2.py:302:9: No overload of function `get_decoded_events_of_transaction` matches arguments
- error[no-matching-overload] rotkehlchen/tests/api/test_eth2.py:1091:12: No overload of function `get_decoded_events_of_transaction` matches arguments
- error[no-matching-overload] rotkehlchen/tests/api/test_eth2.py:1382:5: No overload of function `get_decoded_events_of_transaction` matches arguments
- error[no-matching-overload] rotkehlchen/tests/api/test_ethereum_transactions.py:1422:5: No overload of function `get_decoded_events_of_transaction` matches arguments
- error[no-matching-overload] rotkehlchen/tests/api/test_ethereum_transactions.py:1469:5: No overload of function `get_decoded_events_of_transaction` matches arguments
- error[no-matching-overload] rotkehlchen/tests/api/test_ethereum_transactions.py:1528:9: No overload of function `get_decoded_events_of_transaction` matches arguments
- error[no-matching-overload] rotkehlchen/tests/api/test_sushiswap.py:59:5: No overload of function `get_decoded_events_of_transaction` matches arguments
- error[no-matching-overload] rotkehlchen/tests/api/test_uniswap.py:72:5: No overload of function `get_decoded_events_of_transaction` matches arguments
- error[no-matching-overload] rotkehlchen/tests/data_migrations/test_migrations.py:550:9: No overload of function `get_decoded_events_of_transaction` matches arguments
- warning[possibly-unbound-attribute] rotkehlchen/tests/integration/test_eth2.py:414:96: Attribute `query_withdrawals` on type `Unknown | Blockscout | None` is possibly unbound
- error[no-matching-overload] rotkehlchen/tests/unit/decoders/test_degen.py:33:17: No overload of function `get_decoded_events_of_transaction` matches arguments
- error[no-matching-overload] rotkehlchen/tests/unit/decoders/test_degen.py:78:17: No overload of function `get_decoded_events_of_transaction` matches arguments
- error[no-matching-overload] rotkehlchen/tests/unit/test_evm_balances.py:43:5: No overload of function `get_decoded_events_of_transaction` matches arguments
- Found 1728 diagnostics
+ Found 1715 diagnostics

scikit-learn (https://github.com/scikit-learn/scikit-learn)
-     memo fields = ~539MB
+     memo fields = ~593MB

scipy (https://github.com/scipy/scipy)
- error[unresolved-attribute] scipy/_lib/array_api_extra/tests/conftest.py:33:26: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] scipy/_lib/array_api_extra/tests/conftest.py:205:16: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] scipy/_lib/array_api_extra/tests/conftest.py:215:18: Type `FixtureRequest` has no attribute `param`
- error[unresolved-attribute] scipy/conftest.py:443:10: Type `FixtureRequest` has no attribute `param`
- Found 6624 diagnostics
+ Found 6620 diagnostics
- TOTAL MEMORY USAGE: ~1271MB
+ TOTAL MEMORY USAGE: ~1399MB

@AlexWaygood AlexWaygood removed their request for review June 30, 2025 16:41
@sharkdp
Copy link
Contributor

sharkdp commented Jun 30, 2025

Thank you for working on this! I am planning to take a closer look tomorrow.

It looks like this leads to a lot new false positives on ecosystem projects (see the auto-generated mypy_primer comment). We should probably try to find out what causes them (+ fix them and write a few more test cases here).

@iyakushev
Copy link
Contributor Author

Thank you @sharkdp! I will try to add some test cases following mypy_primer results tomorrow. Is it okay to add them into mdtests?
As for false positives, I think its due to this line. I mostly copied the way bindings did it while getting familiar with the codebase.

@sharkdp
Copy link
Contributor

sharkdp commented Jul 1, 2025

Thank you @sharkdp! I will try to add some test cases following mypy_primer results tomorrow. Is it okay to add them into mdtests?

Yes. Feel free to add some new sections in https://github.com/astral-sh/ruff/blob/main/crates/ty_python_semantic/resources/mdtest/attributes.md

As for false positives, I think its due to this line. I mostly copied the way bindings did it while getting familiar with the codebase.

That particular part will change with #18955 where we basically decide not to track possible unboundness (and with your PR: undeclaredness) anymore for implicit instance attributes. You could maybe try rebasing your branch on top of that already.

@iyakushev iyakushev requested a review from MichaReiser as a code owner July 1, 2025 16:55
@iyakushev iyakushev marked this pull request as draft July 1, 2025 16:59
@iyakushev iyakushev force-pushed the support-decl-only branch from b5033c4 to 6dac224 Compare July 1, 2025 17:33
@iyakushev
Copy link
Contributor Author

@sharkdp Please take a look when available. It seems to me that most of the errors (false positives) in the original implementation come from class instance access to its attributes via self if they have some form of annotation.

class C:
    def __init__(self):
        self.foo: list[int] = []
        
    def bar(self):
        print(self.foo)  # Previously possibly-unbound 

I've added a few mdtest to validate this (and marked them with TODO). There's definitely something wrong; either its beyond the original scope of the issue, or I'm misusing something. Either way, your guidance would be very valuable 😃
I've changed implementation as well, it only checks declarations at the very end if we have no defined bindings.

@sharkdp
Copy link
Contributor

sharkdp commented Jul 2, 2025

Please take a look when available. It seems to me that most of the errors (false positives) in the original implementation come from class instance access to its attributes via self if they have some form of annotation

I've added a few mdtest to validate this (and marked them with TODO).

I don't understand? Non of the new MD tests seems to show some kind of possibly-unbound false positive?

I've changed implementation as well, it only checks declarations at the very end if we have no defined bindings.

I think we should switch that and check declarations first. If we have a declaration, we should prefer it over any bindings.

@iyakushev
Copy link
Contributor Author

Non of the new MD tests seems to show some kind of possibly-unbound false positive?

Sorry for the ambiguity in my previous message. I've tried to replicate the warnings I saw in mypy_primer repos, but after I've pulled the new changes (The PR you mentioned + main), the issue seems to be resolved. I've double checked against the chess repo snippet from primer, but I did not contribute it to the mdtest. The changes in the mdtest are for broader testing.

I've moved the declaration check above bindings and fixed another potential issue with Class obj/instance access to attributes. In my previous attempt instance attributes were leaking into class objects even if they were declared only in __init__.

Could we, perhaps, run mypy_primer once again?

Copy link

codspeed-hq bot commented Jul 4, 2025

CodSpeed WallTime Performance Report

Merging #19048 will not alter performance

Comparing iyakushev:support-decl-only (a6206f1) with main (b6edfbc)

Summary

✅ 7 untouched benchmarks

@iyakushev iyakushev changed the title [ty] Support scoped declarations [ty] Support declarations-only attributes Jul 4, 2025
@iyakushev iyakushev changed the title [ty] Support declarations-only attributes [ty] Support declaration-only attributes Jul 4, 2025
@iyakushev iyakushev marked this pull request as ready for review July 5, 2025 07:56
Copy link
Contributor

@sharkdp sharkdp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much. This looks great. Just a few minor comments.

@iyakushev
Copy link
Contributor Author

Thank you for taking time to review this!
I've pushed the changes addressing your comments. As for duplication, I've moved the code into a closure, if that's ok. The name might be a little off imho.

@sharkdp sharkdp force-pushed the support-decl-only branch from 884b766 to a6206f1 Compare July 7, 2025 10:46
Copy link
Contributor

@sharkdp sharkdp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great — thank you very much!

@sharkdp sharkdp merged commit e0b7f49 into astral-sh:main Jul 7, 2025
35 checks passed
@iyakushev iyakushev deleted the support-decl-only branch July 7, 2025 14:55
UnboundVariable pushed a commit to UnboundVariable/ruff that referenced this pull request Jul 7, 2025
…c_tokens

* 'main' of https://github.com/astral-sh/ruff: (27 commits)
  [ty] First cut at semantic token provider (astral-sh#19108)
  [`flake8-simplify`] Make example error out-of-the-box (`SIM116`) (astral-sh#19111)
  [`flake8-use-pathlib`] Make example error out-of-the-box (`PTH210`) (astral-sh#19189)
  [`flake8-use-pathlib`] Add autofixes for `PTH203`, `PTH204`, `PTH205` (astral-sh#18922)
  [`flake8-type-checking`] Fix syntax error introduced by fix (`TC008`) (astral-sh#19150)
  [`flake8-pyi`] Make example error out-of-the-box (`PYI007`, `PYI008`) (astral-sh#19103)
  Update Rust crate indicatif to 0.18.0 (astral-sh#19165)
  [ty] Add separate CI job for memory usage stats (astral-sh#19134)
  [ty] Add documentation for server traits (astral-sh#19137)
  Rename to `SessionSnapshot`, move unwind assertion closer (astral-sh#19177)
  [`flake8-type-checking`] Make example error out-of-the-box (`TC001`) (astral-sh#19151)
  [ty] Bare `ClassVar` annotations (astral-sh#15768)
  [ty] Re-enable multithreaded pydantic benchmark (astral-sh#19176)
  [ty] Implement equivalence for protocols with method members (astral-sh#18659)
  [ty] Use RHS inferred type for bare `Final` symbols (astral-sh#19142)
  [ty] Support declaration-only attributes (astral-sh#19048)
  [ty] Sync vendored typeshed stubs (astral-sh#19174)
  Update dependency pyodide to ^0.28.0 (astral-sh#19164)
  Update NPM Development dependencies (astral-sh#19170)
  Update taiki-e/install-action action to v2.56.7 (astral-sh#19169)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ecosystem-analyzer ty Multi-file analysis & type inference
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants