Invoke-AzCostManagementQuery often does not return results on first try. Throttling?

[o-l-a-v]

Description

I experience that Invoke-AzCostManagementQuery often does not return results on first try. Seems the Rest API endpoint it uses is unstable. If I retry once or sometimes even twice, the success rate becomes better.

Can Az do something about this? Add some retry logic or something?

Issue script & Debug output

$null = Add-Member -InputObject $Subscription -Force -NotePropertyName 'ConsumptionTwoWeeksAgo' -NotePropertyValue (
    Invoke-AzCostManagementQuery -WhatIf:$false -DatasetAggregation @{
        'totalCost' = @{
            'name'     = [string] 'Cost'
            'function' = [string] 'SUM'
        }
    } -Type 'ActualCost' -Timeframe 'Custom' `
        -Scope ('/subscriptions/{0}' -f $Subscription.'Id') `
        -TimePeriodFrom $StartOfTwoWeeksAgo.ToString('o') `
        -TimePeriodTo  $EndOfTwoWeeksAgo.ToString('o') 6>$null
)

Environment data

PowerShell v7.5.1 x64
Windows 11 24H2 x64

Module versions

Az.Accounts 4.2.0
Az.CostManagement 0.4.0

Error output

No error

[o-l-a-v]

Seems I get throttled. Az should handle that, right? Seems it does not in this case.

{
  "error": {
    "code": "429",
    "message": "Too many requests. Please retry."
  }
}

[o-l-a-v]

Made a workaround using the API directly with Invoke-AzRestMethod, which returns HTTP status code too.

If header x-ms-ratelimit-microsoft.costmanagement-entity-retry-after is present I wait that amount. Else I just wait 5 seconds to try again.

# Invoke cost query with failproofing and throttling handling
do {
    $Results = Invoke-AzRestMethod -Method 'Post' -WhatIf:$false -Path (
        '/subscriptions/{0}/providers/Microsoft.CostManagement/query?api-version=2019-11-01' -f $SubscriptionId
    ) -Payload (
        ConvertTo-Json -Compress -Depth 4 -InputObject @{
            'timePeriod' = @{
                'from' = [string] $From.ToString('o')
                'to'   = [string] $To.ToString('o')
            }
            'dataset'    = @{
                'aggregation' = @{
                    'totalCost' = @{
                        'name'     = [string] 'Cost'
                        'function' = [string] 'SUM'
                    }
                }
            }
            'type'       = [string] 'ActualCost'
            'timeframe'  = [string] 'Custom'
        }
    )
    if ($Results.'StatusCode' -eq 429) {
        Start-Sleep -Seconds ($Results.'Headers'.'x-ms-ratelimit-microsoft.costmanagement-entity-retry-after' ?? 5)
    }
    elseif ($Results.'StatusCode' -ne 200) {
        Throw ('Request returned status code {0} (means "{1}")' -f $Results.'StatusCode', [System.Net.HttpStatusCode]($Results.'StatusCode'))
    }
}
until ($Results.StatusCode -eq 200)

# Return results
(ConvertFrom-Json -InputObject $Results.'Content').'properties'

[isra-fel]

cc @dolauli

[dolauli]

We will retry only when Retry-After instead of x-ms-ratelimit-microsoft.costmanagement-entity-retry-after is set in the 429 response header.

[o-l-a-v]

So the resource provider does not follow Azure RM API guidelines?

Maybe throw an error or warning on 429 without the expected header? Instead of just returning nothing.