[azservicebus] Memory leak when receiving messages

[gkampitakis]

Bug Report

• Package in question azure-sdk-for-go/sdk/messaging/azservicebus
• SDK version v1.5.0
• go version 1.21

• What happened?

We are trying to move away from the deprecated azure sdk for service bus https://github.com/Azure/azure-service-bus-go and use e the new in one of our services. This service is heavily reading on messages from a Service Bus Queue. We are noticing an increasing usage of memory.

Also from profiling data we can see that the Heap size increase is happening at go-ampq code

• What did you expect or want to happen?

I would expect the new SDK to have a different memory footprint (either higher or lower memory consumption) but after some time i was expecting the memory usage of the service to be stable as before and not increasing time over time.

• How can we reproduce it?

Not sure how easy it is, as we need to leave the service running for a couple of days to start seeing the increase in memory usage.

• Anything we should know about your environment.

Data shared above are from staging environment. I am expecting bigger usage on a prod environment with much higher traffic.

Our setup is we are using a number of worker goroutines each one receiving messages from a client.Receiver in parallel.

[jhendrixMSFT]

Looks related/due to Azure/go-amqp#314.

[gkampitakis]

Indeed it does looks related. Thanks for linking it 👍

[jhendrixMSFT]

Would you be able to share any additional info about your app? I've copied the questions from the linked issue.

• are there multiple senders/receivers?
• are the senders/receivers short or long lived?
• what are typical message sizes and how big is the largest message (best guess is fine)?

[gkampitakis]

• are there multiple senders/receivers?

Yes there are. We have a number of producer goroutines that are receiving messages and a number of consumer goroutines that process messages and communicate via a go channel. At the start of the service we create a pool of receivers client.NewReceiverForQueue for receiving messages in parallel.

• are the senders/receivers short or long lived?

They are long lived. At service start up we create one client and from that client we create the needed receivers and senders that are used throughout the life of the service.

• what are typical message sizes and how big is the largest message (best guess is fine)?

The average message size is ~3Kib and the max message size can be up to 256Kib.

[jhendrixMSFT]

Thanks this is helpful.

Is the number of senders/receivers dynamic or fixed? Sounds like it's fixed but I don't want to make assumptions. If it is dynamic, does the number of senders/receivers have a cap and does it ever shrink?

At the go-amqp level, each sender/receiver has its own message buffer, and the buffer size will grow to the largest sent/received message but will never shrink.

If the number of senders/receivers is fixed, then I would expect that the total buffer size would end up being "count senders/receivers * max message size" and level out. I'm thinking about this in the context of the linked issue. It could be that what you're observing is something different.

[gkampitakis]

Is the number of senders/receivers dynamic or fixed? Sounds like it's fixed but I don't want to make assumptions.

That is correct, it's a fixed number.

If the number of senders/receivers is fixed, then I would expect that the total buffer size would end up being "count senders/receivers * max message size" and level out. I'm thinking about this in the context of the linked issue. It could be that what you're observing is something different.

This is what I expected as well. Our service doesn't consume much memory so I would expect even if there was a memory increase at some point it will level out (given the service is receiving stable traffic).

But this is how our memory is trending in a period of 3 weeks (on staging env)

On the profile I shared above, addUnsettled looks like a potential point of leakage. Which just adds a key in a map if this map never get's empty can cause this type of slow leakage, unfortunately though I am not familiar with the go-ampq to understand the path of addUnsettled and deleteUnsettled.