Sync v2-6-stable with v2-6-test to release 2.6.2 #31796
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…#30813) Some of our users raised issues that when extending the image, airflow suddenly started reporting problem with database versions and migration not aplied or out-of-sync. This almost always turns out to be a dependency conflict, that leads to automated downgrate or upgrade of installed airflow version. This is - obviously - undesired (you should be upgrading airflow consciously rather than accidentally). However there is no way to do it implicitly - `pip` might decide to upgrade or downgrade airflow as it sees fit. From the point of view - airflow is just one of the packages and has no special meaning. The only way to "keep" airflow version is to specify it together with other requirements, pinned to the specific version. This PR updates our examples to do this and explains why airflow is added there. There is - of course - another risk that the user will forget to update the version of airflow when they upgrade, however, sinc this is explicit action performed during image extension, it is much easier to diagnose and notice. We also warn the users that they should upgrade when airflow is upgraded. (cherry picked from commit bf6ebe9)
(cherry picked from commit df90317)
…all (#31448) It is important to keep airflow as a dependency with pinned version if you are adding new dependencies. This PR adds it in case of docker-compose documentation and also improves examples in our images, by using AIRFLOW_VERSION variable defined in the images. This way users can write future-proof ways of installind dependencies basing on our examples. Follow-up after #30813 (cherry picked from commit d722fba)
(cherry picked from commit 8062756)
* add unit tests for default_args overriding in task group Signed-off-by: Hussein Awala <[email protected]> * fix overriding default args in nested task groups Signed-off-by: Hussein Awala <[email protected]> * Update airflow/utils/task_group.py Co-authored-by: Ash Berlin-Taylor <[email protected]> --------- Signed-off-by: Hussein Awala <[email protected]> Co-authored-by: Ash Berlin-Taylor <[email protected]> (cherry picked from commit 9e8627f)
Signed-off-by: Hussein Awala <[email protected]> (cherry picked from commit 8891c69)
* Fix typing for POST user endpoint This should raise exception not return it. * fixup! Fix typing for POST user endpoint (cherry picked from commit f7ed878)
The latest release of Python 3.8 and 3.9 have been just released that contain the fix to a security vulnerability backported to those versions: python/cpython#102153 Release notes: * https://www.python.org/downloads/release/python-3817/ * https://www.python.org/downloads/release/python-3917/ The fix improved sanitizing of the URLs and until Python 3.10 and 3.11 get released, we need to add the sanitization ourselves to pass tests on all versions. In order to improve security of airflow users and make the tests work regardless whether the users have latest Python versions released, we add extra sanitisation step to the URL to apply the standard WHATWG specification. (cherry picked from commit 87c5c9f)
* fix-docs * Update BREEZE.rst (cherry picked from commit 8dc895c)
Co-authored-by: Pierre Jeambrun <[email protected]> Co-authored-by: Pankaj Koti <[email protected]> Co-authored-by: Vincent <[email protected]> Co-authored-by: Kaxil Naik <[email protected]> (cherry picked from commit 6053615)
(cherry picked from commit 2be5d3a)
Co-authored-by: Tzu-ping Chung <[email protected]> --------- Co-authored-by: Malthe Borch <[email protected]> Co-authored-by: eladkal <[email protected]> Co-authored-by: Tzu-ping Chung <[email protected]> (cherry picked from commit a03f6cc)
* Reset permission if `access_control` is empty * Check `resource` before call `_revoke_all_stale_permissions` * Fix static checks (cherry picked from commit 2c0c8b8)
* Fix ExternalTaskSensor to work correctly with task groups that have mapped tasks * Add tests for ExternalTaskSensor with task group that have mapped tasks * Fix working for multiple runs * Use tuple_in_condition instead of tuple_ --------- Co-authored-by: Zhyhimont Dmitry <[email protected]> (cherry picked from commit 3c30e54)
Add a broad-strokes description of the security expectations operator should expect. This will get included into https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/index.html I'm no Airflow expert so help definitely welcome. I would like this section to be a bit more prominent rather than hidden away under 'Administration and Deployment', but it looks like the structure was carefully considered in #27235 so this is probably fine. Eventually we could point the 'Security' link on the main pages like https://airflow.apache.org/ to this page. (cherry picked from commit f20c08a)
Add Python 3.11 support Python 3.11 has been released as scheduled on October 25, 2022 and finally, after all dependencies got upgraded - we can support it. --------- Co-authored-by: Tzu-ping Chung <[email protected]> (cherry picked from commit c5597d1)
* Function returns list of dagruns and not query * Changed pytests * Changed all to _start_queued_dagruns * Added comment and fixed tests * Fixed typo (cherry picked from commit 0fd42ff)
Co-authored-by: Jens Scheffler <[email protected]> (cherry picked from commit ec18db1)
(cherry picked from commit f39ccad)
--------- Co-authored-by: Tzu-ping Chung <[email protected]> (cherry picked from commit 471fdac)
(cherry picked from commit a2a9867)
* Validate connection_ids (cherry picked from commit 5cb8ef8)
* spinner added * fixed static checks * isFetching * isFetching replaced with isLoading (cherry picked from commit 584a9f5)
(cherry picked from commit b4ba6b7)
eladkal
requested review from
kaxil,
potiuk,
ashb and
jedcunningham
as code owners
For PRs that are just changing selected providers, there is no need to install and verify all providers, just those affected should be installed (and full verification should only be done when all providers are being built - so generally speaking, when Core PRs are run or when canary builds are run. This PR distinguishes those two cases: * installing and verification when all providers are selected * installing only affected providers when only subset of them are affected Also a bug was found where core file changes did not trigger provider docs build during implementation. This has been fixed (cherry picked from commit 3af22f5)
Removing unicodescv as dependency invites problems when users will use older hive, google, microsoft providers, because they were using unicodecsv, but they did not declare it as dependency (it was a transitive dependency of the "apache-airflow" package). It has been removed in #31693 Unicodecsv has very low footprint so this is not a problem to keep it. The dependency misses license in it's package, therefore we add the licence in our "licences" folder. (cherry picked from commit a853233)
A number of dev scripts have been cherry-picked in a different order from main and that made it quite a bit more difficult to apply all of the cherry-picks consistently. Also we are supporting wider range of Python versions (3.7 - 3.11) than either previous 2.6 or main, so the breeze scripts and tests had to be updated to reflect those differences. Those changes are in dev tooling only, they have no impact on airflow source code.
When CI runs in non-main branch, some provider precommits and provider tests should be disabled. There was a typo that prevented this from happening, also names of the pre-commits changed. (cherry picked from commit bc7e471)
We do not need to run separate build on push and pull request when we are cherry-picking and pushing branch to apache/airflow repo. Those PRs are always made by committers, and they are done as direct push to the apache/airflow repo so they will run build on push (which will be equivalent of canary run). We also do not need to run anything on stable, because stable is the result of merging those changes from test after they get green. (cherry picked from commit 0dddaa0)
This check is really for the developers who are developing on main and there is no need to run it for other branches. (cherry picked from commit da4912b)
Previously we had them defined both in constant and in config.yml. Now just config.yml (cherry picked from commit cab342e)
(cherry picked from commit f6bb474)
|
Should we also bump 2.6.1 -> 2.6.2 in |
Yes, Elad did that in 4743855, but was overriden when syncing dev scripts with main in 9323ab1.
(moving it will cause annoying conflicts, or will have to move others as well) |
|
Should be good |
pierrejeambrun
approved these changes
Jun 10, 2023
ephraimbuddy
approved these changes
Jun 11, 2023
|
We also need to cherry-pick #31839 since the next build will break breeze unit tests due to Let me do it. |
The recent version of pytest released on June 10th introduced change that it will read conftest tests starting in the directory which is current working directory, this causes breeze tests run with working directory being top airflow directory to fail because it's requirements do not contain a few libraries that are used in airflow's conftest.py This change rather than specifying directory at pytest command line, changes working directory for the tests. (cherry picked from commit 3206719)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Time for
2.6.2rc1!