Tail Sampling Processor: Combine Probabilitstic Policy with Other Policy Types

[sarincasm]

Component(s)

processor/tailsampling

Is your feature request related to a problem? Please describe.

I want to apply different sampling percentages (sample rate) based on various conditions. However, there doesn't seem to be a way to do apply probabilistic sampling policy based on some condition.

It would be quite helpful, if we could combine probabilistic policy with other types. For eg:
combining string_attribute policy, we can construct logic such as

if http.route matches 'foo' (string_attribute policy)
then apply sampling_percentage of 0.1

or similarly, if
attribute.foo matches bar
then apply sampling_percentage of 50

and so on.

Describe the solution you'd like

It would be quite helpful, if we could combine probabilistic policy with other types. For eg:
combining string_attribute policy, we can construct logic such as

if http.route matches 'foo' (string_attribute policy)
then apply sampling_percentage of 0.1

or similarly, if
attribute.foo matches bar
then apply sampling_percentage of 50

and so on.

Describe alternatives you've considered

Other option seems to be to use composite policy, though that doesnt allow probabilistic sampling (based on percent of events), and rather only has the option of defining spans/second.

Additional context

No response

[github-actions]

Pinging code owners:

• processor/tailsampling: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

Have you tried the composite policy? From the docs:

composite: Sample based on a combination of above samplers, with ordering and rate allocation per sampler. Rate allocation allocates certain percentages of spans per policy order. For example if we have set max_total_spans_per_second as 100 then we can set rate_allocation as follows
test-composite-policy-1 = 50 % of max_total_spans_per_second = 50 spans_per_second
test-composite-policy-2 = 25 % of max_total_spans_per_second = 25 spans_per_second
To ensure remaining capacity is filled use always_sample as one of the policies

You can also try the "and" policy:

and: Sample based on multiple policies, creates an AND policy

You can then have a string attribute policy and a probabilistic policy as part of the same "and" policy.

[sarincasm]

Thanks for replying.

Maybe I misunderstood something in the composite policy, but it seems it only allows me to set a rate limit based on output spans per second, and not as a percent of actual input volume of events.
That wont be ideal for my use case, because I want the output volume of events to be a percent of input volume without a predefined output rate limit.

However, using and this way certainly looks promising !!

If I define:
(string_attribute policy where http.route == foo) && (probabilistic policy with a sampling_percentage of X),
then,
this should lead to an overall positive sampled decision in X% of cases where the string_attribute sub policy leads to a positive sampled decision, which is exactly what I want.

I'm gonna try this early next week and share the results here.

Thank you!

[sarincasm]

Hello, this works perfectly!

Here is an example, in case anyone finds it useful:

{
  name: noisy-endpoint-policy,
  type: and,
  and:
    {
      and_sub_policy:
        [
          {
            name: service-name-policy,
            type: string_attribute,
            string_attribute:
              { key: service.name, values: [name-of-service] },
          },
          {
            name: route-policy,
            type: string_attribute,
            string_attribute:
              {
                key: http.route,
                values: [/v1/posts/.+],
                enabled_regex_matching: true,
              },
          },
          {
            name: probabilistic-policy,
            type: probabilistic,
            probabilistic: { sampling_percentage: 1 },
          },
        ],
    },
},