Support inserting "collector instance id" as an attribute to spanmetrics metrics

[golopot]

Component(s)

attributes processor, span metrics connector

Is your feature request related to a problem? Please describe.

It is hard to configure spanmetrics to abide by the single writer principle, in a deployment multiple collectors might export the same metric stream which violates the single writer principle and produces inaccurate metric readings downstream.

This can be solved by attaching a collector instance id attribute to the metric stream, ensuring each collector exports different metrics.

Describe the solution you'd like

Two options:

1. attributesprocessor: Inserting "collector instance id" attribute to metric streams using attributesprocessor.
2. spanmetrics: Inserting "collector instance id" attribute to metric streams using spanmetrics connector.

Describe alternatives you've considered

Adding a layer of load balancer collectors with a service.instance.id based routing key. But there are still conflicting metrics writes when you perform a rolling update deployment on collectors. This can be partially solved by setting metrics_expiration for spanmetrics, but still not 100% solved.

Additional context

No response

[iblancasa]

Can you explain exactly what you would like to have implemented as part of a solution?
Doesn't this help https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector#known-limitation-violation-of-the-single-writer-principle?

[golopot]

The link does help, this issue is an attempt to adopt the recommendation:

If multiple pipelines are used, ensure each produces uniquely identified metrics (e.g., inject attributes using a processor)

The following explains the issue in more detail:

Problem

When deploying multiple OpenTelemetry collectors with the SpanMetrics processor in a load-balanced setup, metrics can become inaccurate due to conflicting writes to Prometheus. This violates the single writer principle for metrics.

In a typical deployment with multiple collectors:

1. Traces are load-balanced across multiple collectors
2. Each collector independently processes spans and generates metrics
3. Different collectors maintain separate metric states
4. When writing to Prometheus, metrics fluctuate between different collector states
5. This leads to inaccurate metric readings in Prometheus

Example deployment:

flowchart TD
    app -->|trace_a, trace_b| loadbalancer
    loadbalancer -->|trace_a| collector-spanmetrics1 
    loadbalancer -->|trace_b| collector-spanmetrics2
    collector-spanmetrics1 -->|"spanmetrics.calls{...}"=100| prometheus
    collector-spanmetrics2 -->|"spanmetrics.calls{...}"=200| prometheus

Loading

Proposed Solution

Add collector identification to metrics exported by the SpanMetrics processor to make metric streams distinct. This can be achieved by adding a colletor_id label to all metrics generated by the SpanMetrics processor, the value should be a stable, unqiue identifier for each collector instance, such as a UUID generated at collector startup.

Expected output

With this change exported metris would look like:

spanmetrics.calls{..., service_name="app1", collector_id="uuidfirst"}
spanmetrics.calls{..., service_name="app1", collector_id="uuidsecond"}

Implementation Details

The solution can be implemented by extending the attributesprocessor configuration to support a special context value collector.id:

processors:
  attributes/addcollectorid:
    actions:
      - key: collector_id
        action: insert
        from_context: collector.id # A special context value 

And applying it in the pipeline:

service:
  extensions: [health_check]
    pipelines:
      m:
        receivers: [spanmetrics]
        processors: [attributes/addcollectorid]
        exporters: [otlp/prometheus]
    ...

At prometheus query side the metrics can then be aggregated as:

sum by (service_name) (spanmetrics_calls) 

-> 

spanmetrics_calls {service_name="app1"}

This restores the desired metric.

[iblancasa]

What do you propose to use as ID? Because this can be tricky depending on the env… because using the UUID can be a problem in envs where collector restarts are common (and will increase cardinality a lot.

[github-actions]

Pinging code owners for processor/attributes: @boostchicken. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[golopot]

Yes the cardinality might be a problem. But I cannot think of a better ID than a generated UUID at startup.

[golopot]

mention spanmetrics code owners @portertech, @Frapschen

[iblancasa]

Yes the cardinality might be a problem. But I cannot think of a better ID than a generated UUID at startup.

That’s the reason why you should select the atrributes depending on your use case as the provided link describes. UUID is not a good solution.

[golopot]

That link does not suggest any concrete attribute, so I am here.

My use case is to setup a single otlp gateway for traces ingestion (no sidecar collectors) that also generates spanmetrics metrics. The collectors deployment need to be scalabe. Do you have any recommemdation on how to setup such a deployment?

My current setup involves a loadbalancerexporter layer that routes traces with a routing_key based on service.instance.id . But this setup has a conflict metrics writer problem when you add new collector instances to the deployment, because the loadbalancerexporter routing is not sticky on backend deployment changes.

[iblancasa]

Because that will depend on the attributes your telemetry data contains. You should follow what the last line says:

Instead, use attributes that are stable, present in all spans, and meaningfully distinguish each stream. Good examples include cluster_id, region, or deployment_environment.

You can generate an UUID, for instance, and inject it via env var.

[cforce]

For opamp managed collectors the collector.id is stored persistent. At collector startup the same id will reused if already existing (file) so the collector/supervisor is not seen as new but reconnected. I would like to to attach this id to all telemetry as resource attribute as well which is not implemented yet