Description
I know it is eaarly (Python 3.11 has just been released yesterday) but we are hoping in Apache Airflow to a much faster cycle of adding new Python releases - especially that Pyhon 3.11 introduces huge performance improvements (25% is the average number claimed) due to a very focused effort to increase single-threaded Python performance (Specialized interpreter being the core of it but also many other improvements) without actually changing any of the Python code.
The google provider will be a huge drag on Airlfow's compatibility for Python 3.11 and we might even decide to release Airlfow without Google Provider support for 3.11. Though it would be great to avoid that.
Google Provider (as it was originally mentioned in #12116) has still a number of old google cloud libraries < 2.0.0 that for sure will not get good. Also support for libraries such as bigquery for Python 3.11 has to be added (but this is external to it and tracked in googleapis/python-bigquery#1386)
Nice summary of Py3.11 support is here: https://pyreadiness.org/3.11/ - it's not very green obviously, but I hope it gets greener soon.
I just opened the PR to add 3.11 support yesterday and plan to keep it open until it gets green :)
I think it would be fantastic if we could work out all the problems and add migrate all the old dependencies:
- google-api-python-client>=1.6.0,<2.0.0
- google-cloud-bigtable>=1.0.0,<2.0.0 Remove <2.0.0 limit on google-cloud-bigtable #29644
- google-cloud-dlp>=0.11.0,<2.0.0 Update google cloud dlp package and adjust hook and operators #29234
- google-cloud-language>=1.1.1,<2.0.0
- google-cloud-secret-manager>=0.2.0,<2.0.0
- google-cloud-spanner>=1.10.0,<2.0.0
- google-cloud-speech>=0.36.3,<2.0.0
- google-cloud-texttospeech>=0.4.0,<2.0.0
- google-cloud-translate>=1.5.0,<2.0.0
- google-cloud-videointelligence>=1.7.0,<2.0.0
- google-cloud-vision>=0.35.2,<2.0.0
- python-bigquery-sqlalchemy is a blocker for PROD images: Support for Python 3.11 googleapis/python-bigquery-sqlalchemy#500
Looking forward to cooperation on that one :)
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.