Bump azure-kusto-data>=4.1.0

[pankajastro]

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[uranusjr]

Wow that’s a big leap. Would it be reasonable/meaningful to include some of the versions between 4.1 and 1.0?

[pankajastro]

Wow that’s a big leap. Would it be reasonable/meaningful to include some of the versions between 4.1 and 1.0?

I want to add support for DefaultAzureCredential look like it will work with >4.1.0 https://github.com/Azure/azure-kusto-python/releases/tag/v4.1.0

[Lee-W]

Wow that’s a big leap. Would it be reasonable/meaningful to include some of the versions between 4.1 and 1.0?

I want to add support for DefaultAzureCredential look like it will work with >4.1.0 https://github.com/Azure/azure-kusto-python/releases/tag/v4.1.0

Or would it be reasonable to enable this feature only for users who's using azure-kusto-data>=4.1.0?

[phanikumv]

Wow that’s a big leap. Would it be reasonable/meaningful to include some of the versions between 4.1 and 1.0?

I want to add support for DefaultAzureCredential look like it will work with >4.1.0 https://github.com/Azure/azure-kusto-python/releases/tag/v4.1.0

Or would it be reasonable to enable this feature only for users who's using azure-kusto-data>=4.1.0?

yeah i feel that is more reasonable.

[potiuk]

Wow that’s a big leap. Would it be reasonable/meaningful to include some of the versions between 4.1 and 1.0?
yeah i feel that is more reasonable.

Comment:

I don't have, any problems with big jumps like that to be honest - especially in providers. IMHO - if we need a new functionality from a new version of dependency and we are already (via constraints) using newer version of it, this is no brainer and we should simply bump the min version - even if it is few major versions.

We're doing it quite often when we see that we want to add a new feature or make sure we do not want to trigger som bugs. For example I just 2 days ago Bumped Celery to 5.3.0+ - relatively new releease from Jun - because it unblocked some other dependencies and actually allowed us to upgrade opentelemetry to newer versions (we were held back before).

I think there are very little risks involved in bumping individual dependencies with their minimum versions. For 9X% of our users they are anyhow already using constraints, and if we are bumping the min version to somethign "below" our constraints, it's rather "safe" bet.

Of course there might be edge cases that users won't be able to upgrade because they are somehow tied to older versions of the dependency, but:

a) risk of it is really small

b) effect of it is that simply won't be able upgrade to a new version of provider (but if they are somehow linked to the old version they will know that during the installation and even - if they properly follow our best practices of ours, the provider will be automatically downgraded to the right version by pip resolver when they install older version of the dependency - they might even not notice that they downgrade happened.

c) they can still use the old version of provider for a long time according to our policies even if they upgrade to a new version of airflow (basically - forever).

d) the problem with keeping min-version for a long time is that we do not actually KNOW if the min version is right. We might have added a new feature or incompatible code, accidentally and might have not realised that because we are using the latest "matching" version of the dependency in our tests. So by bumping min version we are actually making it MORE likely that someone will avoid problems (which they might have by using older version that we broke compatibility with accidentallly)

So for me I see no reason at all to avoid such jump. IMHO This is completely no brainer that we should bump min version unconditionally here if we want to use the new feature.

[pankajastro]

Thanks @potiuk for the details explanation.

Or would it be reasonable to enable this feature only for users who's using azure-kusto-data>=4.1.0

@Lee-W @phanikumv I feel this would not be good from a maintenance point of view.

[potiuk]

Also one more good tihng about bumping the dependency like that. When you have too many versions to consider, pip might have hard ttime to resolve what is the best set of dependencies - this is what happened with (extreme case) > 300 versions of botocore + related packages we had, which caused pip to backtrack - #33649. - so by bumping them, we make airflow installation work faster (and possibly eventually avoid backtracking).

[pankajastro]

@potiuk wondering if need to increase the timeout for ARM image build job https://github.com/apache/airflow/blob/main/.github/workflows/build-images.yml#L242 job getting timeout after 50 min I tried couple of times

[potiuk]

@potiuk wondering if need to increase the timeout for ARM image build job https://github.com/apache/airflow/blob/main/.github/workflows/build-images.yml#L242 job getting timeout after 50 min I tried couple of times

I think we lost capability of using the ARM hardware for the image build in the last upgrade. Let me see.

[potiuk]

Yep. This is the reason:

ERROR: failed to initialize builder airflow_cache (airflow_cache1): Cannot connect to the Docker daemon at tcp://localhost:12357. Is the docker daemon running?

[potiuk]

Oh yeah. I was lazy in the past and did not create a separate autossh installation script and now removed it accidentally when I switched to a newer version of docker. apache/airflow-ci-infra@d3600b6#diff-68e4a1b74366f7a6d636f986d2b2b68feba4776d9074d4e2132498d8ae0b2909L44

[potiuk]

When you rebase next time, I think it should work with the ARM instance. I updated AMI with autossh and when the ARM job will pick up the new AMI it should work fast