Unexpected primitive based Enum (de)serialization

[mixilchenko]

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When enumeration inherits some primitive type (e.g. str, int), deserialization makes the value primitive as well.

import enum
from airflow.serialization.serde import deserialize, serialize

class MyEnum(enum.IntEnum):
    A = 1
    B = 2

value = MyEnum.A
res = deserialize(serialize(value))
assert res is MyEnum.A, f'{type(res)=}, {res=}'
# AssertionError: type(res)=<class 'int'>, res=1

What you think should happen instead?

This is little unexpected for me as a user and for mypy

@task
def my_task(v: MyEnum):
    assert is instance(v, MyEnum), 'fine for mypy, fails in runtime'
    v = MyEnum(v)
...
my_task(MyEnum.A)

The other issue is that if I add airflow compatible custom serialization the result is incorrect too

class MyEnum(enum.IntEnum):
    A = 1
    B = 2

    def serialize(self):
        return f'{self.name}.{self.value}'

res = serialize(MyEnum.A)
assert res == 'A.1', f'{type(res)=}, {res=}'
# AssertionError: type(res)=<class 'int'>, res=1

I would expect one of

• implement default serialization and deserialization for enums
• fail with TypeError: cannot serialize object of type <enumeration 'MyEnum'> unless it implements the interface and is registered for deserialization (as it happens for enums that do not inherit primitive)

How to reproduce

See the snippet above

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[amoghrajesh]

@mixilchenko i think we need to have a serializer + deserializer that does this properly for Enum types. Would you wanna work on this?

[Lcrs123]

The other issue is that if I add airflow compatible custom serialization the result is incorrect too

I've come across this problem before and was thinking about opening an issue. If your custom type inherits from a python primitive, the sequence of checks that airflow does will always result in it using the de/serialize methods for the primitive type, even if you have defined custom de/serialize methods in your class yourself.

Maybe the order of checks should be changed? If the developer went through the trouble of defining the custom methods himself, he obviously wants them to be used, and not the default ones.

[amarlearning]

Hi @amoghrajesh, I would like to work on this issue :)