[Bug] GenConverter: Wrong resolving / missing __parameters__ attribute when inheriting something from `typing.*`

[raabf]

• cattrs version: 1.10.0
• Python version: 3.7
• Operating System: Linux 5.16.1-1 OpenSUSE Thumbleweed

There is an exception AttributeError: type object 'CLASS' has no attribute '__parameters__' when you try to structure an attrs Class which inherts typing.CLASS

Reproduce

Have a look at this example where CLASS is Hashable:

import attr
import typing
import cattrs
import collections

converter = cattrs.GenConverter()

@attr.define
class A(collections.abc.Hashable):  # typing.Hashable should be preferred over collections.abc.Hashable
    attr1: str = ''
    attr2: int = 0

    def __hash__(self) -> int:
        return hash(self.attr2)

@attr.define
class B(typing.Hashable):
    attr1: str = ''
    attr2: int = 0

    def __hash__(self) -> int:
        return hash(self.attr2)


def main() -> None:
    converter.structure(dict(attr1='hello', attr2=42), A)  # OK
    converter.structure(dict(attr1='hello', attr2=42), B)  # AttributeError: type object 'Hashable' has no attribute '__parameters__'

if __name__ == '__main__':
    main()

Output

Traceback (most recent call last):
  File "/home/raabf/sl/demo.py", line 40, in <module>
    main()
  File "/home/raabf/sl/demo.py", line 37, in main
    converter.structure(dict(attr1='hello', attr2=42), B)  # AttributeError: type object 'Hashable' has no attribute '__parameters__'
  File "/home/raabf/.pyenv/versions/sl37/lib/python3.7/site-packages/cattr/converters.py", line 300, in structure
    return self._structure_func.dispatch(cl)(obj, cl)
  File "/home/raabf/.pyenv/versions/sl37/lib/python3.7/site-packages/cattr/dispatch.py", line 49, in _dispatch
    return self._function_dispatch.dispatch(cl)
  File "/home/raabf/.pyenv/versions/sl37/lib/python3.7/site-packages/cattr/dispatch.py", line 126, in dispatch
    return handler(typ)
  File "/home/raabf/.pyenv/versions/sl37/lib/python3.7/site-packages/cattr/converters.py", line 745, in gen_structure_attrs_fromdict
    **attrib_overrides,
  File "/home/raabf/.pyenv/versions/sl37/lib/python3.7/site-packages/cattr/gen.py", line 224, in make_dict_structure_fn
    mapping = _generate_mapping(base, mapping)
  File "/home/raabf/.pyenv/versions/sl37/lib/python3.7/site-packages/cattr/gen.py", line 195, in _generate_mapping
    for p, t in zip(get_origin(cl).__parameters__, get_args(cl)):
AttributeError: type object 'Hashable' has no attribute '__parameters__'

Note that in the above code snippet I use typing.Hashable just as an example, the same exception occurs if you inherit something other from typing.*, such as typing.Reversable or typing.Iterable.

Offending Part

The offending part in cattrs is:

https://github.com/python-attrs/cattrs/blob/v1.10.0/src/cattr/gen.py#L206-L225

def make_dict_structure_fn(
    cl: Type[T],
    converter: "Converter",
    _cattrs_forbid_extra_keys: bool = False,
    _cattrs_use_linecache: bool = True,
    _cattrs_prefer_attrib_converters: bool = False,
    **kwargs,
) -> Callable[[Mapping[str, Any]], T]:
    """Generate a specialized dict structuring function for an attrs class."""

    mapping = {}
    if is_generic(cl):
        base = get_origin(cl)
        mapping = _generate_mapping(cl, mapping)
        cl = base

    for base in getattr(cl, "__orig_bases__", ()):
        if is_generic(base) and not str(base).startswith("typing.Generic"):
            mapping = _generate_mapping(base, mapping)
            break

which again calls

def _generate_mapping(
    cl: Type, old_mapping: Dict[str, type]
) -> Dict[str, type]:
    mapping = {}
    for p, t in zip(get_origin(cl).__parameters__, get_args(cl)):
        if isinstance(t, TypeVar):
            continue
        mapping[p.__name__] = t

    if not mapping:
        return old_mapping

    return mapping

where __parameters__ is accessed. During the exception cl is typing.Hashable, which resolves to collections.abc.Hashable by get_origin(cl).

Reason

The use of typing.Hashable is correct. As far as I understand, during static analyisis from tools such as mypy it acts as type checking, during runtime it is resolved to collections.abc.Hashable where the class is implemented.
I personally rely on inheriting from typing.Hashable instead of collections.abc.Hashable, or else mypy would do wrong type checking.

I do not understand what cattrs is doing, so I just can make some guesses:

• The resolving from typing.Hashable to collections.abc.Hashable at runtime (i.e. typing.Hashable has collections.abc.Hashable as its origin) confuses cattrs.
• Some check before fails and _generate_mapping should have been never called with typing.Hashable
• In general you cannot assume that all origins have the __parameters__ attribute, i.e. use getattr(get_origin(cl), '__parameters__', ()) instead.

Reference for getting parameters

The library typing_inspect has an get_parameters function to get __parameters__. The function is a bit longer and handles some corner cases , I do not know if this helps with problem reported in this issue here or can be utilized here.

[Tinche]

Interesting.

Before I dive deeper into this, since I had a Python 3.11 virtual env activated I tried your code in that and it worked. So this might've gotten fixed in Python at some point? 3.8 seems to also error, 3.9 works.

[raabf]

Can confirm, did not expect a different bahaviour on different python versions, since I thought this is something by design.

• 3.7.12 ↦ Error occurs
• 3.8.12 ↦ Error occurs
• 3.9.10 ↦ No Error

However, I am not sure what exactly got ‘fixed‘ in python 3.9. collections.abc.Hashable still does not have a __parameters__ attribute, but the error does not occur any more because _generate_mapping is not called any more. That again is caused because is_generic(base) in if is_generic(base) and not str(base).startswith("typing.Generic"): of the function make_dict_structure_fn returns False since python 3.9.

[Tinche]

Yeah, this is development using Python runtime types in a nutshell ;) If you look around on the issue tracker, you'll find an issue introduced between 3.9.9 and 3.9.10.

get_args from typing also has its idiosyncrasies between versions. If we can fix this issue with getattr(get_origin(cl), '__parameters__', ()) I'm very open to merging that in since it's a super minimal change.

[raabf]

Using getattr(get_origin(cl), '__parameters__', ()) as it makes _generate_mapping a no-op if the __parameters__ do not exist. And It seems to fix my current problem. Hence I made #221 .

is_generic

Still as a follow up question I must ask if cattrs._compat.is_generic() is a correct implementation?

I compared that with typing_inspection.is_generic_type Which always returns True:

Python 3.7.12 (default, Nov 12 2021, 17:41:59)
[GCC 11.2.1 20210816 [revision 056e324ce46a7924b5cf10f61010cf9dd2ca10e9]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing_inspect import is_generic_type
>>> from typing import Hashable 
>>> is_generic_type(Hashable)
True
>>> from cattr._compat import is_generic
>>> is_generic(Hashable)
True

Python 3.9.10 (main, Feb  2 2022, 19:30:01) 
[GCC 11.2.1 20220103 [revision d4a1d3c4b377f1d4acb34fe1b55b5088a3f293f6]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing_inspect import is_generic_type
>>> from typing import Hashable
>>> is_generic_type(Hashable)
True
>>> from cattr._compat import is_generic
>>> is_generic(Hashable)
False

So typing_inspect.is_generic_type() is always True, although there is no special handling betwenn the python versions, but cattrs._compat.is_generic() behaves differently.

[raabf]

Beside that I thought about it, and I think #221 is correct behaviour, because if a class is generic it must not neccessary have __parameters__.
For example

>>> import typing
>>> class A(typing.Generic[T]):                                                                                      
...         ...                                                                                                            
...
>>> class B(A[str]):
...         ...                                                                                                            
...
>>> A.__parameters__
(~T,)
>>> B.__parameters__
()

With B.__parameters__ there is no AttributeError like on accessing typing.Hashable.__parameters__ but the list is still empty and hence _generate_mapping would be in effect a no-op.

[Tinche]

Hm can this be closed?

[raabf]

Yea thanks, as far as I see now, the MR #221 should handle all cases where a Generic is not complete, i.e. does not have an __parameters__.