Problem with Microsoft Azure Provider base_azure hook

[shakeelansari63]

Apache Airflow version

2.6.3

What happened

When using AzureBaseHook from airflow/providers/microsoft/azure/hooks/base_azure.py with SDKClient class, the get_conn() method fails to instanciate the Client class object.

The error message says, missing parameter credential

This is because the sdk_client is being instanciated with credentials=ServicePrincipalCredentials

Notice an extra s in credentials key which is not needed.

What you think should happen instead

The sdk_client should be instanciated with credential=ServicePrincipalCredential which will be instanciated correctly

How to reproduce

1. Install any Azure client, let's say azure-monitor-query
   pip install azure-monitor-query

2. Use the Azure connection with Azure client

from airflow.providers.microsoft.azure.hooks.base_azure import AzureBaseHook
from azure.monitor.query import LogsQueryClient

# Lets say we have an azure connection with name Azure_Default

conn = AzureBaseHook(sdk_client = LogsQueryClient, conn_id = 'Azure_Default')
client = conn.get_conn()

This will fail -

Operating System

Any

Versions of Apache Airflow Providers

apache-airflo-providers-microsoft-azure v6.2.1

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[nathadfield]

@shakeelansari63 This looks like a simple change but I have not been able to replicate this with the code you provided. Can you review this or do you have an example DAG that can illustrate the problem?

[shakeelansari63]

Hi @nathadfield ,

Here is example -
I have created an Azure Connection As Follow -

Note: I haven't added any key_json or key_path in extra. Which means, following piece of code will be ran to instanciate the client -

And When I call the get_conn() method, it gives following error -

And this error is because it need credentail not credentials.

[nathadfield]

@shakeelansari63 Having made this change, we are now encountering test failures and so I remain unconvinced that this is indeed a problem.

        self.log.info("Getting connection using specific credentials and subscription_id.")
>       return self.sdk_client(
            credential=ServicePrincipalCredentials(client_id=conn.login, secret=conn.password, tenant=tenant),
            subscription_id=subscription_id,
        )
E       TypeError: __init__() got an unexpected keyword argument 'credential'

[nathadfield]

Changing this seems to break AzureContainerInstanceHook.

[pankajkoti]

I added a comment on the PR guessing that could be the cause for failing tests #33040 (comment)

[shakeelansari63]

@nathadfield , I got the issue here.
Looks like you are using very old version of azure SDK.

I looked into one of the failure, for ContainerInstanceManagementClient.

This is from provider.yaml

 - azure-mgmt-containerinstance>=1.5.0,<2.0

This is very old version of azure SDK (from 2019) which is still using credentials parameter. But in newer version all azure sdks use credential parameter.

[shakeelansari63]

From what I can see from Pypi, latest version of azure-mgmt-containerinstance is 10.1.0

[nathadfield]

@shakeelansari63 Yep. I'll ask around to see what I can find out but it seems like you won't be able to use LogsQueryClient until azure-mgmt-containerinstance is brought up to date.

[nathadfield]

#30199

[eladkal]

I'm closing this issue as the root cause is updating old package as listed in #30199

PRs are welcome.
Azure offers Airflow as managed service in their eco system try maybe contacting Azure support maybe they will assign an engineer to work on Azure open source issues.