[ISSUE] Issue with provider azure auth wrong audience/scope for management token

[shortpoet]

Configuration

• this first conf is suddenly broken (but was working for quite some time)

$env:ARM_SUBSCRIPTION_ID  = "..."
$env:ARM_TENANT_ID= "..."
$env:ARM_CLIENT_ID= "..."

data "databricks_sql_warehouse" "warehouse" {
  count = local.enable_databricks_integration ? 1 : 0

  name = local.resource_databricks_sql_warehouse_name
}

provider "databricks" {
  host = local.enable_databricks_integration ? data.azurerm_databricks_workspace.workspace[0].workspace_url : null
}

• this is a workaround that i would rather avoid

$env:ARM_SUBSCRIPTION_ID  = "..."
$env:ARM_TENANT_ID= "..."
$env:ARM_CLIENT_ID= "..."

data "databricks_sql_warehouse" "warehouse" {
  count = local.enable_databricks_integration ? 1 : 0

  name = local.resource_databricks_sql_warehouse_name
}

provider "databricks" {
  host = local.enable_databricks_integration ? data.azurerm_databricks_workspace.workspace[0].workspace_url : null
  azure_client_id     = var.provider_databricks_azure_client_id
  azure_client_secret = var.provider_databricks_azure_client_secret
  azure_tenant_id     = local.tenant_id_lazlo
}

Expected Behavior

Should have been able to fetch the data source with provided creds and no need for additional databricks provider configuration

Actual Behavior

• When looking at logs and the token used, I ran the same API call using that token and got invalid audience error

• Expected aud claim to be: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d, but was: https://management.core.windows.net/.",

• terraform error with log file parsed

 Error: cannot read sql warehouse: cannot read data sql warehouse: unable to parse response. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log: │ ``` │ GET /api/2.0/preview/sql/data_sources │ > * Host:  │ > * Accept: application/json │ > * Authorization: REDACTED │ > * Traceparent: 00-e44a5c6d76d3d49af804f3f6e0f0e827-c34d6bd77d08f6c0-01 │ > * User-Agent: databricks-tf-provider/1.62.0 databricks-sdk-go/0.54.0 go/1.22.10 os/windows terraform/1.10.2 sdk/sdkv2 data/yes resource/sql_warehouse auth/azure-client-secret │ > * X-Databricks-Azure-Sp-Management-Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImltaTBZMnowZFlLeEJ0dEFxS19UdDVoWUJUayIsImtpZCI6ImltaTBZMnowZFlLeEJ0dEFxS19UdDVoWUJUayJ9.eyJhdWQiOiJodHRwczovL21hbmFnZW1lbnQuY29yZS53aW5kb3dzLm5ldC8iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC85NGI1N2ZlNy1kZjc3LTQ4ODYtOGE1Yi0zYTc5M2YzNWMzNDkvIiwiaWF0IjoxNzM5Mzk0NDIwLCJuYmYiOjE3MzkzOTQ0MjAsImV4cCI6MTczOTM5ODMyMCwiYWlvIjoiazJSZ1lFajAvUFRzMXJXYVEySWZ1ekpQdHE0OUNnQT0iLCJhcHBpZCI6ImRiNmMyNDZiLWY1NGQtNGNhZC1iMzI4LWRkZjU5NDY5ZTgzYSIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0Lzk0YjU3ZmU3LWRmNzctNDg4Ni04YTViLTNhNzkzZjM1YzM0OS8iLCJpZHR5cCI6ImFwcCIsIm9pZCI6IjU0ODc1MjQxLTdmZWItNDlkNi05NTI2LTQxNGQ5MmJjNmIwMCIsInJoIjoiMS5BUnNBNTMtMWxIZmZoa2lLV3pwNVB6WERTVVpJZjNrQXV0ZFB1a1Bhd2ZqMk1CUFlBQUFiQUEuIiwic3ViIjoiNTQ4NzUyNDEtN2ZlYi00OWQ2LTk1MjYtNDE0ZDkyYmM2YjAwIiwidGlkIjoiOTRiNTdmZTctZGY3Ny00ODg2LThhNWItM2E3OTNmMzVjMzQ5IiwidXRpIjoiRVJDcnpyTnJnRUNuVGNraW83UnFBQSIsInZlciI6IjEuMCIsInhtc19pZHJlbCI6IjE4IDciLCJ4bXNfdGNkdCI6MTQ3ODIxMzAwOH0.tHhnZDRZDI7r155Hx_F_RFm8l3exsYduAZcS2Xyy3zYrEkGmtxpaT0yTjb5rYhAFCIV4ldOpqzMs5Elh_tDk27KiE6NuVzwxAva1C6-kmTWnU_C0zTOLZtjF5caWX-G-sqa1D3QHPq4c7Yx52gh3yn7GOJQgliJVj63S05WOoZJPG-3l1b6WY0qXyF6-HZ-A4LdrG_cZgU_x-CHlpbElAxRvzXxiV9n7mUgDKGK5l9fES858Wb9kiiYQoHBCDyX0WprMaPX3HZ_VOp1xcmBZJi5pTjsTvamLen3A8Q3Y7rGGZ0-eGVv_SAN9ajU2iKO8qYsU_tscHn9B_m3rjSDj7Q │ < HTTP/2.0 403 Forbidden │ < * Content-Length: 20 │ < * Content-Type: text/html; charset=utf-8 │ < * Date: Wed, 12 Feb 2025 21:12:00 GMT │ < * Server: databricks │ < * X-Request-Id: 7042de2b-b387-9d81-a4fd-00ac46e89d3c │ < User not authorized. │ ``` │  │   with data.databricks_sql_warehouse.warehouse[0], │   on datasources.tf line 181, in data "databricks_sql_warehouse" "warehouse": │  181: data "databricks_sql_warehouse" "warehouse" { │  ╵

##[section] [Get-TerraformErrors] Resolving Terraform Errors

##[section] Initializing Terraform Error Map

##[section] [Get-TerraformErrors] No Terraform Errors in Environment Variable - Using Log File

##[section] Initializing Terraform Error Map

##[section] [Get-TerraformNetworkError] Resolving Network Error:
vertex "data.databricks_sql_warehouse.warehouse[0]" error: cannot read sql warehouse: cannot read data sql warehouse: unable to parse response. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:
GET /api/2.0/preview/sql/data_sources
> * Host:
> * Accept: application/json
> * Authorization: REDACTED
> * Traceparent: 00-e44a5c6d76d3d49af804f3f6e0f0e827-c34d6bd77d08f6c0-01
> * User-Agent: databricks-tf-provider/1.62.0 databricks-sdk-go/0.54.0 go/1.22.10 os/windows terraform/1.10.2 sdk/sdkv2 data/yes resource/sql_warehouse auth/azure-client-secret
> * X-Databricks-Azure-Sp-Management-Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImltaTBZMnowZFlLeEJ0dEFxS19UdDVoWUJUayIsImtpZCI6ImltaTBZMnowZFlLeEJ0dEFxS19UdDVoWUJUayJ9.eyJhdWQiOiJodHRwczovL21hbmFnZW1lbnQuY29yZS53aW5kb3dzLm5ldC8iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC85NGI1N2ZlNy1kZjc3LTQ4ODYtOGE1Yi0zYTc5M2YzNWMzNDkvIiwiaWF0IjoxNzM5Mzk0NDIwLCJuYmYiOjE3MzkzOTQ0MjAsImV4cCI6MTczOTM5ODMyMCwiYWlvIjoiazJSZ1lFajAvUFRzMXJXYVEySWZ1ekpQdHE0OUNnQT0iLCJhcHBpZCI6ImRiNmMyNDZiLWY1NGQtNGNhZC1iMzI4LWRkZjU5NDY5ZTgzYSIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0Lzk0YjU3ZmU3LWRmNzctNDg4Ni04YTViLTNhNzkzZjM1YzM0OS8iLCJpZHR5cCI6ImFwcCIsIm9pZCI6IjU0ODc1MjQxLTdmZWItNDlkNi05NTI2LTQxNGQ5MmJjNmIwMCIsInJoIjoiMS5BUnNBNTMtMWxIZmZoa2lLV3pwNVB6WERTVVpJZjNrQXV0ZFB1a1Bhd2ZqMk1CUFlBQUFiQUEuIiwic3ViIjoiNTQ4NzUyNDEtN2ZlYi00OWQ2LTk1MjYtNDE0ZDkyYmM2YjAwIiwidGlkIjoiOTRiNTdmZTctZGY3Ny00ODg2LThhNWItM2E3OTNmMzVjMzQ5IiwidXRpIjoiRVJDcnpyTnJnRUNuVGNraW83UnFBQSIsInZlciI6IjEuMCIsInhtc19pZHJlbCI6IjE4IDciLCJ4bXNfdGNkdCI6MTQ3ODIxMzAwOH0.tHhnZDRZDI7r155Hx_F_RFm8l3exsYduAZcS2Xyy3zYrEkGmtxpaT0yTjb5rYhAFCIV4ldOpqzMs5Elh_tDk27KiE6NuVzwxAva1C6-kmTWnU_C0zTOLZtjF5caWX-G-sqa1D3QHPq4c7Yx52gh3yn7GOJQgliJVj63S05WOoZJPG-3l1b6WY0qXyF6-HZ-A4LdrG_cZgU_x-CHlpbElAxRvzXxiV9n7mUgDKGK5l9fES858Wb9kiiYQoHBCDyX0WprMaPX3HZ_VOp1xcmBZJi5pTjsTvamLen3A8Q3Y7rGGZ0-eGVv_SAN9ajU2iKO8qYsU_tscHn9B_m3rjSDj7Q
< HTTP/2.0 403 Forbidden
< * Content-Length: 20
< * Content-Type: text/html; charset=utf-8
< * Date: Wed, 12 Feb 2025 21:12:00 GMT
< * Server: databricks
< * X-Request-Id: 7042de2b-b387-9d81-a4fd-00ac46e89d3c
< User not authorized.

• my script output

.\Utils\Invoke-AuthRunner.ps1 -Command run-dapi -Params 'Get-Warehouse lpr qa' $t
Vault SecretStore requires a password.
Enter password:
**************
##[info] [Assert-LoginToken] Token -> eyJ0eXAi ... 1311 ... CD5S4r1g -> Copied to clipboard
##[info] [Invoke-AuthRunner] {
  "login_necessary": null,
  "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImltaTBZMnowZFlLeEJ0dEFxS19UdDVoWUJUayIsImtpZCI6ImltaTBZMnowZFlLeEJ0dEFxS19UdDVoWUJUayJ9.eyJhdWQiOiJodHRwczovL21hbmFnZW1lbnQuY29yZS53aW5kb3dzLm5ldC8iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC85NGI1N2ZlNy1kZjc3LTQ4ODYtOGE1Yi0zYTc5M2YzNWMzNDkvIiwiaWF0IjoxNzM5MzkyNTUyLCJuYmYiOjE3MzkzOTI1NTIsImV4cCI6MTczOTM5NjQ1MiwiYWlvIjoiazJSZ1lMRGF2Wk1sdnBmdGttNTh3d1dIdHpwckFRPT0iLCJhcHBpZCI6ImRiNmMyNDZiLWY1NGQtNGNhZC1iMzI4LWRkZjU5NDY5ZTgzYSIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0Lzk0YjU3ZmU3LWRmNzctNDg4Ni04YTViLTNhNzkzZjM1YzM0OS8iLCJpZHR5cCI6ImFwcCIsIm9pZCI6IjU0ODc1MjQxLTdmZWItNDlkNi05NTI2LTQxNGQ5MmJjNmIwMCIsInJoIjoiMS5BUnNBNTMtMWxIZmZoa2lLV3pwNVB6WERTVVpJZjNrQXV0ZFB1a1Bhd2ZqMk1CUFlBQUFiQUEuIiwic3ViIjoiNTQ4NzUyNDEtN2ZlYi00OWQ2LTk1MjYtNDE0ZDkyYmM2YjAwIiwidGlkIjoiOTRiNTdmZTctZGY3Ny00ODg2LThhNWItM2E3OTNmMzVjMzQ5IiwidXRpIjoiRU1WeVZsc2c1a3lzWkp2WjgwQThBQSIsInZlciI6IjEuMCIsInhtc19pZHJlbCI6IjcgMjQiLCJ4bXNfdGNkdCI6MTQ3ODIxMzAwOH0.a7uSN7wkbylRPJaQ61WdQ97cwiRhIfFPTSFrGNQgEyd4u3qkPWOFquylXSUYQBFYoUiY4Tk7l24jsLNFGVJX19F8h3s-oYl3D7rdkJK2dYvl9DS3rCaPN0PL_P-NCEz90a37EqOHeYhcv0QwOQEreqhqYfIATTxMLydu-3f5syGCOLTEFG756Ao5SX0IPM1SIxHCW7nY6Ts1UH0TJh45PPK6W4KLcOBMZi1zhBme6tH-yf_ivCMjxPM--IgXenbJ7onsEGELQmkyY2XjBcxFvzJFi1sWQ4iTlTkUmMfGUTnONih1-ijbJCcJs_WHROXmv10ktBpfll0_zGCD5S4r1g"
}
##[info] [Invoke-AuthRunner] {
  "Token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImltaTBZMnowZFlLeEJ0dEFxS19UdDVoWUJUayIsImtpZCI6ImltaTBZMnowZFlLeEJ0dEFxS19UdDVoWUJUayJ9.eyJhdWQiOiJodHRwczovL21hbmFnZW1lbnQuY29yZS53aW5kb3dzLm5ldC8iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC85NGI1N2ZlNy1kZjc3LTQ4ODYtOGE1Yi0zYTc5M2YzNWMzNDkvIiwiaWF0IjoxNzM5MzkyNTUyLCJuYmYiOjE3MzkzOTI1NTIsImV4cCI6MTczOTM5NjQ1MiwiYWlvIjoiazJSZ1lMRGF2Wk1sdnBmdGttNTh3d1dIdHpwckFRPT0iLCJhcHBpZCI6ImRiNmMyNDZiLWY1NGQtNGNhZC1iMzI4LWRkZjU5NDY5ZTgzYSIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0Lzk0YjU3ZmU3LWRmNzctNDg4Ni04YTViLTNhNzkzZjM1YzM0OS8iLCJpZHR5cCI6ImFwcCIsIm9pZCI6IjU0ODc1MjQxLTdmZWItNDlkNi05NTI2LTQxNGQ5MmJjNmIwMCIsInJoIjoiMS5BUnNBNTMtMWxIZmZoa2lLV3pwNVB6WERTVVpJZjNrQXV0ZFB1a1Bhd2ZqMk1CUFlBQUFiQUEuIiwic3ViIjoiNTQ4NzUyNDEtN2ZlYi00OWQ2LTk1MjYtNDE0ZDkyYmM2YjAwIiwidGlkIjoiOTRiNTdmZTctZGY3Ny00ODg2LThhNWItM2E3OTNmMzVjMzQ5IiwidXRpIjoiRU1WeVZsc2c1a3lzWkp2WjgwQThBQSIsInZlciI6IjEuMCIsInhtc19pZHJlbCI6IjcgMjQiLCJ4bXNfdGNkdCI6MTQ3ODIxMzAwOH0.a7uSN7wkbylRPJaQ61WdQ97cwiRhIfFPTSFrGNQgEyd4u3qkPWOFquylXSUYQBFYoUiY4Tk7l24jsLNFGVJX19F8h3s-oYl3D7rdkJK2dYvl9DS3rCaPN0PL_P-NCEz90a37EqOHeYhcv0QwOQEreqhqYfIATTxMLydu-3f5syGCOLTEFG756Ao5SX0IPM1SIxHCW7nY6Ts1UH0TJh45PPK6W4KLcOBMZi1zhBme6tH-yf_ivCMjxPM--IgXenbJ7onsEGELQmkyY2XjBcxFvzJFi1sWQ4iTlTkUmMfGUTnONih1-ijbJCcJs_WHROXmv10ktBpfll0_zGCD5S4r1g"
}
Token provided, using it
DapiToken: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImltaTBZMnowZFlLeEJ0dEFxS19UdDVoWUJUayIsImtpZCI6ImltaTBZMnowZFlLeEJ0dEFxS19UdDVoWUJUayJ9.eyJhdWQiOiJodHRwczovL21hbmFnZW1lbnQuY29yZS53aW5kb3dzLm5ldC8iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC85NGI1N2ZlNy1kZjc3LTQ4ODYtOGE1Yi0zYTc5M2YzNWMzNDkvIiwiaWF0IjoxNzM5MzkyNTUyLCJuYmYiOjE3MzkzOTI1NTIsImV4cCI6MTczOTM5NjQ1MiwiYWlvIjoiazJSZ1lMRGF2Wk1sdnBmdGttNTh3d1dIdHpwckFRPT0iLCJhcHBpZCI6ImRiNmMyNDZiLWY1NGQtNGNhZC1iMzI4LWRkZjU5NDY5ZTgzYSIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0Lzk0YjU3ZmU3LWRmNzctNDg4Ni04YTViLTNhNzkzZjM1YzM0OS8iLCJpZHR5cCI6ImFwcCIsIm9pZCI6IjU0ODc1MjQxLTdmZWItNDlkNi05NTI2LTQxNGQ5MmJjNmIwMCIsInJoIjoiMS5BUnNBNTMtMWxIZmZoa2lLV3pwNVB6WERTVVpJZjNrQXV0ZFB1a1Bhd2ZqMk1CUFlBQUFiQUEuIiwic3ViIjoiNTQ4NzUyNDEtN2ZlYi00OWQ2LTk1MjYtNDE0ZDkyYmM2YjAwIiwidGlkIjoiOTRiNTdmZTctZGY3Ny00ODg2LThhNWItM2E3OTNmMzVjMzQ5IiwidXRpIjoiRU1WeVZsc2c1a3lzWkp2WjgwQThBQSIsInZlciI6IjEuMCIsInhtc19pZHJlbCI6IjcgMjQiLCJ4bXNfdGNkdCI6MTQ3ODIxMzAwOH0.a7uSN7wkbylRPJaQ61WdQ97cwiRhIfFPTSFrGNQgEyd4u3qkPWOFquylXSUYQBFYoUiY4Tk7l24jsLNFGVJX19F8h3s-oYl3D7rdkJK2dYvl9DS3rCaPN0PL_P-NCEz90a37EqOHeYhcv0QwOQEreqhqYfIATTxMLydu-3f5syGCOLTEFG756Ao5SX0IPM1SIxHCW7nY6Ts1UH0TJh45PPK6W4KLcOBMZi1zhBme6tH-yf_ivCMjxPM--IgXenbJ7onsEGELQmkyY2XjBcxFvzJFi1sWQ4iTlTkUmMfGUTnONih1-ijbJCcJs_WHROXmv10ktBpfll0_zGCD5S4r1g

##[section] Start of script
##[debug] Workspace URL:         adb-5246339893763581.1.azuredatabricks.net
##[debug] Dapi token:            eyJ0eXAi
##[debug] Command:               Get-Warehouse
##[debug] Param:
GET-ing -> https://adb-5246339893763581.1.azuredatabricks.net/api/2.0/sql/warehouses
##[info]
Request method: GET
Request URI: https://adb-5246339893763581.1.azuredatabricks.net/api/2.0/sql/warehouses
Error: io.jsonwebtoken.IncorrectClaimException: Expected aud claim to be: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d, but was: https://management.core.windows.net/.
res: {
  "Exception": {
    "errorMessage": "Response status code does not indicate success: 400 (Bad Request).",
    "responseMessage": "io.jsonwebtoken.IncorrectClaimException: Expected aud claim to be: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d, but was: https://management.core.windows.net/.",
    "responseCode": 400,
    "TargetSite": null,
    "Message": "io.jsonwebtoken.IncorrectClaimException: Expected aud claim to be: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d, but was: https://management.core.windows.net/.",
    "Data": {},
    "InnerException": null,
    "HelpLink": null,
    "Source": null,
    "HResult": -2146233088,
    "StackTrace": null

Steps to Reproduce

Debug Output

Important Factoids

Would you like to implement a fix?