Description
Component(s)
connector/spanmetrics
Is your feature request related to a problem? Please describe.
OTEP 235 describes how to encode sampling probability, and now probabilistic sampler processor supports it.
I propose to two new boolean flags to the Config of this component:
sampling_adjustment(default: false) When disabled, each span counts 1. When enabled and sampling has been recorded, each span counts as the inverse of its sampling probability.fractional_counting(default: false) When disabled, spans are counted as integer data points. When enabled, spans are counted as floating point number data points. _Note this only applies to Sum points, not to Histogram point count fields, because OpenTelemetry does not (currently) support floating-point count histograms.
When the sampling adjustment feature is enabled and the fractional counting feature is disabled, there is a potential for errors to be introduced stemming from either inadequate precision or from the use of non-integer-reciprocal sampling probabilities.
As an example of the first case:
The sampler is configured with 33.33% sampling, which is sufficiently close to 1-in-3 that integer counts will have very small error using the threshold calculated by pkg/sampling. However, the sampler is also configured with sampling_precision: 1 which forces the effective probability down in this case. Note the rejection threshold ot=th:a equals 10/16 = 37.5%, and the rejection threshold ot=th:b equals 11/16 = 31.25%. The sampler will output ot=th:b in this case, and the effective adjustment equals exactly 1/(1 - 11/16) = 3.2, which rounds down to 3 for a error of 6.7%. The user should raise sampling precision to lower the systematic error.
As an example of the second case:
The sampler is configured for 75% sampling. This is exactly expressed using powers-of-two, and the adjustment in this case is 1.333. No amount of precision will help in this case. The user should choose sampling probabilities that equate with integer counts. This rules out sampling percentages above 50%.
Describe the solution you'd like
When a sampling adjustment is used without fractional counting, a warning will be issued for spans with sampling probability with an unacceptable margin of error.
Describe alternatives you've considered
When a sampling adjustment is used without fractional counting, a floating-point valued metric named M_residue will be incremented (for metric named M) with the residual error. This amount can be monitored and used to correct the integer-valued metric.