Span Metrics connector support for OTEP 235 probability sampling

[jmacd]

Component(s)

connector/spanmetrics

Is your feature request related to a problem? Please describe.

OTEP 235 describes how to encode sampling probability, and now probabilistic sampler processor supports it.

I propose to two new boolean flags to the Config of this component:

• sampling_adjustment (default: false) When disabled, each span counts 1. When enabled and sampling has been recorded, each span counts as the inverse of its sampling probability.
• fractional_counting (default: false) When disabled, spans are counted as integer data points. When enabled, spans are counted as floating point number data points. _Note this only applies to Sum points, not to Histogram point count fields, because OpenTelemetry does not (currently) support floating-point count histograms.

When the sampling adjustment feature is enabled and the fractional counting feature is disabled, there is a potential for errors to be introduced stemming from either inadequate precision or from the use of non-integer-reciprocal sampling probabilities.

As an example of the first case:

The sampler is configured with 33.33% sampling, which is sufficiently close to 1-in-3 that integer counts will have very small error using the threshold calculated by pkg/sampling. However, the sampler is also configured with sampling_precision: 1 which forces the effective probability down in this case. Note the rejection threshold ot=th:a equals 10/16 = 37.5%, and the rejection threshold ot=th:b equals 11/16 = 31.25%. The sampler will output ot=th:b in this case, and the effective adjustment equals exactly 1/(1 - 11/16) = 3.2, which rounds down to 3 for a error of 6.7%. The user should raise sampling precision to lower the systematic error.

As an example of the second case:

The sampler is configured for 75% sampling. This is exactly expressed using powers-of-two, and the adjustment in this case is 1.333. No amount of precision will help in this case. The user should choose sampling probabilities that equate with integer counts. This rules out sampling percentages above 50%.

Describe the solution you'd like

When a sampling adjustment is used without fractional counting, a warning will be issued for spans with sampling probability with an unacceptable margin of error.

Describe alternatives you've considered

When a sampling adjustment is used without fractional counting, a floating-point valued metric named M_residue will be incremented (for metric named M) with the residual error. This amount can be monitored and used to correct the integer-valued metric.

Additional context

open-telemetry/semantic-conventions#793

[github-actions]

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

@portertech @Frapschen please review

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[portertech]

@iblancasa Top of mind. Please take look 🙏

[jmacd]

@jamesmoessis could you summarize the status of this issue?

[jamesmoessis]

Hi @jmacd apologies but since this discussion has happened my priorities have changed and I don't think I'll get time to work on this. I'll unassign myself for now. I'm also probably not the best guy to summarise the discussion so far as I haven't been apart of sampling SIG discussions relating to above

[iblancasa]

Where can I get some info about this? I would like to work on this.

[atoulme]

Assigned to you @iblancasa