Distribution value with |count| of 0 has a non-zero |mean| value

[gruffT]

Describe the bug
We see a large number of error logs with the same message of form Distribution value with |count| of 0 has a non-zero |mean| value of XXX.
I believe this is connected to #4868 which was recently resolved.

com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[135]: Field timeSeries[135].points[0].distributionValue had an invalid value: Distribution value with |count| of 0 has a non-zero |mean| value of 92.655747445859873.
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1307)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1070)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:819)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:568)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:569)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
		at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
		at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
		at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1908)
		at io.micrometer.stackdriver.StackdriverMeterRegistry.publish(StackdriverMeterRegistry.java:219)
		at io.micrometer.core.instrument.push.PushMeterRegistry.publishSafelyOrSkipIfInProgress(PushMeterRegistry.java:64)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
		at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
		... 3 more
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[135]: Field timeSeries[135].points[0].distributionValue had an invalid value: Distribution value with |count| of 0 has a non-zero |mean| value of 92.655747445859873.
	at io.grpc.Status.asRuntimeException(Status.java:532)
	... 14 more

Environment

• Micrometer version 1.14.4
• Micrometer registry micrometer-registry-stackdriver (GCP)
• OS: GCP Cloud Run, Linux (Redhat I believe)
• Java version: 17

To Reproduce
How to reproduce the bug:

Expected behavior
No emit lots of error logs.

Additional context
Add any other context about the problem here, e.g. related issues.

Thank you for resolving #4868, I can confirm we saw those errors stop with application of the patch.

[shakuzen]

@gruffT Thanks for the report. This should be fixed in the latest snapshots. Would you be able to test with snapshots to let us know if you have any other issues? With the integration tests in #5944, hopefully we can avoid such issues in the future.

[gruffT]

Hi @shakuzen , Sorry for the delay in getting back to you. We've released micrometer-core-1.15.0-SNAPSHOT.jar with micrometer-registry-stackdriver-1.15.0-M2.jar but we're still seeing these errors pretty consistently in our logs I'm afraid.

Any further suggestions gratefully received.

com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[128]: Field timeSeries[128].points[0].distributionValue had an invalid value: Distribution value with |count| of 0 has a non-zero |mean| value of 89.0435405529197.
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1307)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1070)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:819)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:568)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:569)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
		at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
		at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
		at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1908)
		at io.micrometer.stackdriver.StackdriverMeterRegistry.publish(StackdriverMeterRegistry.java:219)
		at io.micrometer.core.instrument.push.PushMeterRegistry.publishSafelyOrSkipIfInProgress(PushMeterRegistry.java:64)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
		at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
		... 3 more
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[128]: Field timeSeries[128].points[0].distributionValue had an invalid value: Distribution value with |count| of 0 has a non-zero |mean| value of 89.0435405529197.
	at io.grpc.Status.asRuntimeException(Status.java:532)
	... 14 more

[izeye]

with micrometer-registry-stackdriver-1.15.0-M2.jar

@gruffT Looking at the changes made for this, to verify the fix, you need to use a snapshot version for the micrometer-registry-stackdriver dependency.

[shakuzen]

Yes, the fix was in micrometer-registry-stackdriver, but generally it is best to use a consistent version for all Micrometer dependencies in case the fix or other changes that may affect things working are across different modules. This can be done with the micrometer-bom or other forms of dependency management such as the dependency management offered by Spring Boot. You'll want to use that to override the version of all micrometer dependencies to e.g. 1.15.0-SNAPSHOT. Could you try that and let us know if you still see any issues? Thanks for the effort on this. It's appreciated.

[gruffT]

Aha, of course. Sorry my bad. I can confirm that seems to have fixed the issue. Thanks so much for the fast response. 👍

[shakuzen]

@gruffT Thank you for reporting the issue and testing the fix so quickly. The fix will be in the next round of releases.