CPU usage increases ostensibly with update to grpc 1.66.0

[atoulme]

Component(s)

No response

Describe the issue you're reporting

See screenshots and current load tests. https://open-telemetry.github.io/opentelemetry-collector-contrib/benchmarks/loadtests/

I see a 10% increase in CPU usage in load test on update to grpc 1.66.0.

I also see a modest increase in memory usage.

This change was introduced with #34978

[atoulme]

@mx-psi @songy23 you merged the change that caused this increase, please take a look.

[mx-psi]

@atoulme Are we sure this increase has been sustained over time? I am not sure how to look at the full historic data for those benchmarks, but given the fact that this commit does not change any Collector code (just bumps the dependencies) and the time that has passed since the change I would like to make sure this was not fixed in grpc-go/its dependencies since.

[songy23]

Also thanks @atoulme for catching and raising this, although I'm more concerned on how to detect this in the future. Right now we only run load tests on mainline commits and it seems already too much (#38446). I assume we don't want to run load tests in every PR, that means we cannot know how a PR impacts performance before merging it. On top of that I don't think there is any alert on the load test numbers and it needs a human to check the graphs from time to time which obviously won't work out in the long run.

[atoulme]

@songy23 please open issues to provide alerts regarding load tests.

@mx-psi please open issues also regarding historical lookback.

What about the issue at hand?

[mx-psi]

@atoulme Pressumably you have been able to have access to the historical data on this, could you clarify how before I file any issues?