[azservicebus] Exceeding max entity size is a retryable error

[eccles]

Bug Report

import "github.com/Azure/azure-sdk-for-go/sdk/messaging/azservicebus"

SDK version: from go.mod
github.com/Azure/azure-sdk-for-go v66.0.0+incompatible
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.1.4
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.1.0
github.com/Azure/azure-sdk-for-go/sdk/messaging/azservicebus v1.1.4
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.4.1
github.com/Azure/go-autorest/autorest v0.11.28
github.com/Azure/go-autorest/autorest/azure/auth v0.5.12
github.com/Azure/go-autorest/autorest/date v0.3.0

    github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2 // indirect
    github.com/Azure/go-autorest v14.2.0+incompatible // indirect
    github.com/Azure/go-autorest/autorest/adal v0.9.21 // indirect
    github.com/Azure/go-autorest/autorest/azure/cli v0.4.5 // indirect
    github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect
    github.com/Azure/go-autorest/autorest/validation v0.3.0 // indirect
    github.com/Azure/go-autorest/logger v0.2.1 // indirect
    github.com/Azure/go-autorest/tracing v0.6.0 // indirect
    github.com/AzureAD/microsoft-authentication-library-for-go v0.7.0 // indirect

• output of go version
  go version go1.20.1 linux/amd64

• What happened?

  Microservice kept crashing when attempting to send message of 1074024736 bytes. The code attempted to send message 3 time with backoffs which is incorrect because the error returned is not retryable. Consequently our code kept retrying instead of handling the error immediately.

• What did you expect or want to happen?

  Immediate suitable error return indicating non-retryable

• How can we reproduce it?

  The call is

    err = c.sender.SendMessage(ctx, message, nil)
    if err != nil {
       ...
  

• Anything we should know about your environment.

N/A

Here is the actual log output:

2023-04-17T09:42:59.613Z DEBUG azbus/message.go:18 msg context before setting timeout: context.Background
2023-04-17T09:42:59.613Z DEBUG azbus/message.go:24 context deadline from message lock deadline: 2023-04-17 09:43:59.582 +0000 UTC
2023-04-17T09:42:59.613Z DEBUG azservicebus@v1.1.4/receiver.go:607 [azsb.Receiver] [lnGAhPjycjDj-dnZV_1RbOMv1qR5e3Ig-PvxneLp8JUrDOnaRl4MPA] Message releaser starting...
2023-04-17T09:42:59.614Z INFO b-newassetprocessorv1/newassetprocessor.go:292 processing asset commit for tenant 0x1e7845E9b3edeB63eb556291E98E11EB58B3369e: assets/596634c5-60c5-40d4-9060-6adb25132c11
2023-04-17T09:42:59.709Z INFO b-newassetprocessorv1/newassetprocessor.go:352 re-publishing 0x1e7845E9b3edeB63eb556291E98E11EB58B3369e:AssetCreator assets/596634c5-60c5-40d4-9060-6adb25132c11 NewAsset
2023-04-17T09:42:59.709Z DEBUG azbus/pubsub.go:203 Send message ethevents (properties=map[processor_apiname:AssetCreator processor_partition_pub:xeqTV2bnvRRKOwa0zxdv9E574Zs2P/tCErqlBuSofmw= processor_topic:eventsenderv1topic proof_mechanism:simple_hash])
2023-04-17T09:43:00.984Z DEBUG utils/retrier.go:69 [azsb.Sender] (SendMessage) Resetting retry attempts
2023-04-17T09:43:00.984Z DEBUG utils/retrier.go:88 [azsb.Sender] (SendMessage) Retry attempt -1 returned retryable error: link detached, reason: *Error{Condition: amqp:resource-limit-exceeded, Description: The maximum entity size has been reached or exceeded for Topic: 'JITAVIDB8D785C8F19A8FEE:TOPIC:ETHEVENTS'. Size of entity in bytes:1074024736, Max entity size in bytes: 1073741824. For more information please see https://aka.ms/ServiceBusExceptions . QuotaType: EntitySize Reference:ab57ccb0-0df0-4945-86c4-c3b91b59c0ea, TrackingId:30e546c6000094aa03572e75643d14a3_G53_B3, SystemTracker:jitavidb8d785c8f19a8fee:Topic:ethevents, Timestamp:2023-04-17T09:43:00, Info: map[]}
2023-04-17T09:43:02.205Z DEBUG utils/retrier.go:88 [azsb.Sender] (SendMessage) Retry attempt 0 returned retryable error: link detached, reason: *Error{Condition: amqp:resource-limit-exceeded, Description: The maximum entity size has been reached or exceeded for Topic: 'JITAVIDB8D785C8F19A8FEE:TOPIC:ETHEVENTS'. Size of entity in bytes:1074024736, Max entity size in bytes: 1073741824. For more information please see https://aka.ms/ServiceBusExceptions . QuotaType: EntitySize Reference:44fdbac8-a65e-47a2-a1a3-bd9eb4b77fd4, TrackingId:30e546c6000094aa03572ed7643d14a5_G53_B3, SystemTracker:jitavidb8d785c8f19a8fee:Topic:ethevents, Timestamp:2023-04-17T09:43:02, Info: map[]}
2023-04-17T09:43:02.205Z DEBUG utils/retrier.go:53 [azsb.Sender] (SendMessage) Retry attempt 1 sleeping for 3.840839713s
2023-04-17T09:43:07.400Z DEBUG utils/retrier.go:88 [azsb.Sender] (SendMessage) Retry attempt 1 returned retryable error: link detached, reason: *Error{Condition: amqp:resource-limit-exceeded, Description: The maximum entity size has been reached or exceeded for Topic: 'JITAVIDB8D785C8F19A8FEE:TOPIC:ETHEVENTS'. Size of entity in bytes:1074024736, Max entity size in bytes: 1073741824. For more information please see https://aka.ms/ServiceBusExceptions . QuotaType: EntitySize Reference:2c1de0cf-ba17-40b2-ac9f-4b4be58c7032, TrackingId:30e546c6000094aa03572fda643d14aa_G53_B3, SystemTracker:jitavidb8d785c8f19a8fee:Topic:ethevents, Timestamp:2023-04-17T09:43:07, Info: map[]}
2023-04-17T09:43:07.400Z DEBUG utils/retrier.go:53 [azsb.Sender] (SendMessage) Retry attempt 2 sleeping for 8.313161578s
2023-04-17T09:43:16.985Z DEBUG utils/retrier.go:88 [azsb.Sender] (SendMessage) Retry attempt 2 returned retryable error: link detached, reason: *Error{Condition: amqp:resource-limit-exceeded, Description: The maximum entity size has been reached or exceeded for Topic: 'JITAVIDB8D785C8F19A8FEE:TOPIC:ETHEVENTS'. Size of entity in bytes:1074024736, Max entity size in bytes: 1073741824. For more information please see https://aka.ms/ServiceBusExceptions . QuotaType: EntitySize Reference:95b59990-0d0c-4643-b8c5-5df49dc9e609, TrackingId:30e546c6000094aa0357319e643d14b3_G53_B3, SystemTracker:jitavidb8d785c8f19a8fee:Topic:ethevents, Timestamp:2023-04-17T09:43:16, Info: map[]}
2023-04-17T09:43:16.985Z DEBUG utils/retrier.go:53 [azsb.Sender] (SendMessage) Retry attempt 3 sleeping for 21.867040651s
2023-04-17T09:43:40.110Z DEBUG utils/retrier.go:88 [azsb.Sender] (SendMessage) Retry attempt 3 returned retryable error: link detached, reason: *Error{Condition: amqp:resource-limit-exceeded, Description: The maximum entity size has been reached or exceeded for Topic: 'JITAVIDB8D785C8F19A8FEE:TOPIC:ETHEVENTS'. Size of entity in bytes:1074024736, Max entity size in bytes: 1073741824. For more information please see https://aka.ms/ServiceBusExceptions . QuotaType: EntitySize Reference:35869351-5164-47d8-965b-ddb0289bc6df, TrackingId:30e546c6000094aa035735fa643d14ca_G53_B3, SystemTracker:jitavidb8d785c8f19a8fee:Topic:ethevents, Timestamp:2023-04-17T09:43:40, Info: map[]}
2023-04-17T09:43:40.110Z INFO azbus/pubsub.go:220 Send failed in 40.400987754s: (connlost): link detached, reason: *Error{Condition: amqp:resource-limit-exceeded, Description: The maximum entity size has been reached or exceeded for Topic: 'JITAVIDB8D785C8F19A8FEE:TOPIC:ETHEVENTS'. Size of entity in bytes:1074024736, Max entity size in bytes: 1073741824. For more information please see https://aka.ms/ServiceBusExceptions . QuotaType: EntitySize Reference:35869351-5164-47d8-965b-ddb0289bc6df, TrackingId:30e546c6000094aa035735fa643d14ca_G53_B3, SystemTracker:jitavidb8d785c8f19a8fee:Topic:ethevents, Timestamp:2023-04-17T09:43:40, Info: map[]}
2023-04-17T09:43:40.110Z DEBUG azbus/receiver.go:95 Reschedule Message context='context.Background.WithDeadline(2023-04-17 09:43:59.582 +0000 UTC [19.471644325s]).WithValue(type opentracing.contextKey, val ).WithValue(type opentracing.contextKey, val )'
2023-04-17T09:43:40.110Z INFO azbus/receiver.go:96 Reschedule Message err='(connlost): link detached, reason: *Error{Condition: amqp:resource-limit-exceeded, Description: The maximum entity size has been reached or exceeded for Topic: 'JITAVIDB8D785C8F19A8FEE:TOPIC:ETHEVENTS'. Size of entity in bytes:1074024736, Max entity size in bytes: 1073741824. For more information please see https://aka.ms/ServiceBusExceptions . QuotaType: EntitySize Reference:35869351-5164-47d8-965b-ddb0289bc6df, TrackingId:30e546c6000094aa035735fa643d14ca_G53_B3, SystemTracker:jitavidb8d785c8f19a8fee:Topic:ethevents, Timestamp:2023-04-17T09:43:40, Info: map[]}'
2023-04-17T09:43:40.110Z DEBUG azbus/pubsub.go:252 Processing message took 40.496481884s
2023-04-17T09:43:40.110Z FATAL startup/startup.go:14 service failed with an error: (connlost): link detached, reason: *Error{Condition: amqp:resource-limit-exceeded, Description: The maximum entity size has been reached or exceeded for Topic: 'JITAVIDB8D785C8F19A8FEE:TOPIC:ETHEVENTS'. Size of entity in bytes:1074024736, Max entity size in bytes: 1073741824. For more information please see https://aka.ms/ServiceBusExceptions . QuotaType: EntitySize Reference:35869351-5164-47d8-965b-ddb0289bc6df, TrackingId:30e546c6000094aa035735fa643d14ca_G53_B3, SystemTracker:jitavidb8d785c8f19a8fee:Topic:ethevents, Timestamp:2023-04-17T09:43:40, Info: map[]}

[eccles]

Please note that the attempt to send an oversized message is expected to fail - just not in a retryable way.

[eccles]

It would be very useful if the customer could determine programmatically whether an error is transient or retryable.

Func IsRetryable(err)

or

func IsTransient()

If I am trying to send a message that is too large then I must simply throw it away - it is not even possible to deadletter it.

[eccles]

FYI I upgraded to latest version of Azure packages but it made no difference:

    github.com/Azure/azure-sdk-for-go v68.0.0+incompatible
    github.com/Azure/azure-sdk-for-go/sdk/azcore v1.4.0
    github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.2
    github.com/Azure/azure-sdk-for-go/sdk/messaging/azservicebus v1.2.1
    github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.4.1
    github.com/Azure/go-autorest/autorest v0.11.28
    github.com/Azure/go-autorest/autorest/azure/auth v0.5.12
    github.com/Azure/go-autorest/autorest/date v0.3.0

[richardpark-msft]

Hi @eccles, this seems pretty reasonable. At the very least we shouldn't be retrying on what is clearly a fatal error, as you've noted.