When running multiple collector instances, each instance generates the same metric.

[ixiaoyi93]

Component(s)

No response

What happened?

Description

When running Collector in multi-instance gateway mode, each Collector instance generates a metric, taking runtime_jvm_gc_timeOrCount_v2 as an example

4 instances of the opentelemetry collector are running.

Steps to Reproduce

Run multiple Collector instances.

Expected Result

Only one metric is generated for that pod or host.

Actual Result

In multiple collectors, the number of corresponding nodes will be generated.

Collector version

contrib 0.114.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: collector-gateway
  namespace: grafana-stack
  labels:
    app.kubernetes.io/name: gateway
    app.kubernetes.io/instance: collector-gateway
    app.kubernetes.io/version: "0.115.1"
    app.kubernetes.io/component: standalone-collector
data:
  relay: |
    connectors:
      servicegraph:
        latency_histogram_buckets:
        - 100ms
        - 200ms
        - 500ms
        - 1s
        - 5s
        - 10s
        metrics_flush_interval: 30s
        store:
          max_items: 10
          ttl: 1s
      spanmetrics:
        aggregation_temporality: AGGREGATION_TEMPORALITY_CUMULATIVE
        dimensions:
        - name: http.request.method
        - name: http.response.status_code
        - name: http.route
        dimensions_cache_size: 1000
        events:
          dimensions:
          - name: exception.type
          - name: exception.message
          enabled: true
        exemplars:
          enabled: true
        histogram:
          explicit:
            buckets:
            - 100us
            - 1ms
            - 2ms
            - 6ms
            - 10ms
            - 100ms
            - 250ms
        metrics_expiration: 5m
        metrics_flush_interval: 15s
        resource_metrics_key_attributes:
        - service.name
        - telemetry.sdk.language
        - telemetry.sdk.name
    exporters:
      debug:
        verbosity: detailed
      otlp:
        endpoint: tempo-distributor:4317
        tls:
          insecure: true
      prometheus:
        add_metric_suffixes: false
        enable_open_metrics: true
        endpoint: 0.0.0.0:8889
        resource_to_telemetry_conversion:
          enabled: true
    extensions:
      health_check:
        endpoint: ${env:MY_POD_IP}:13133
    processors:
      batch:
        send_batch_size: 10000
        timeout: 0s
      filter:
        error_mode: ignore
        traces:
          span:
          - IsMatch(attributes["http.route"], ".*/health.*")
      k8sattributes:
        passthrough: true
      memory_limiter:
        check_interval: 5s
        limit_percentage: 80
        spike_limit_percentage: 25
      transform:
        error_mode: ignore
        trace_statements:
          - context: resource
            statements:
              - set(attributes["env"], attributes["k8s.namespace.name"])
              - replace_match(attributes["env"], "*inner", "inner")
              - replace_match(attributes["env"], "*prod", "prod")
              - replace_all_patterns(attributes, "key", "k8s\\.namespace\\.name", "namespace")
              - replace_all_patterns(attributes, "key", "k8s\\.container\\.name", "container")
              - replace_all_patterns(attributes, "key", "k8s\\.node\\.name", "nodename")
              - replace_all_patterns(attributes, "key", "k8s\\.pod\\.ip", "podip")
              - replace_all_patterns(attributes, "key", "k8s\\.pod\\.name", "podname")
        metric_statements:
          - context: resource
            statements:
              - set(attributes["env"], attributes["k8s.namespace.name"])
              - replace_match(attributes["env"], "*inner", "inner")
              - replace_match(attributes["env"], "*prod", "prod")
              - replace_all_patterns(attributes, "key", "k8s\\.namespace\\.name", "namespace")
              - replace_all_patterns(attributes, "key", "k8s\\.container\\.name", "container")
              - replace_all_patterns(attributes, "key", "k8s\\.node\\.name", "nodename")
              - replace_all_patterns(attributes, "key", "k8s\\.pod\\.ip", "podip")
              - replace_all_patterns(attributes, "key", "k8s\\.pod\\.name", "podname")
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:4317
          http:
            endpoint: ${env:MY_POD_IP}:4318
      prometheus:
        config:
          scrape_configs:
            - job_name: opentelemetry-collector-gateway
              scrape_interval: 10s
              metric_relabel_configs:
                - source_labels: [server]
                  regex: "unknown"
                  action: drop
              static_configs:
                - targets:
                  - ${env:MY_POD_IP}:8888
    service:
      extensions:
      - health_check
      pipelines:
        traces:
          exporters:
          - otlp
          - spanmetrics
          - servicegraph
          - debug
          processors:
          - k8sattributes
          - transform
          - filter
          - memory_limiter
          - batch
          receivers:
          - otlp
        metrics:
          exporters:
          - prometheus
          processors:
          - k8sattributes
          - transform
          - filter
          - memory_limiter
          - batch
          receivers:
          - otlp
        metrics/servicegraph:
          exporters:
          - prometheus
          processors:
          - k8sattributes
          - transform
          - filter
          - memory_limiter
          - batch
          receivers:
          - servicegraph
        metrics/spanmetrics:
          exporters:
          - prometheus
          processors:
          - k8sattributes
          - transform
          - filter
          - memory_limiter
          - batch
          receivers:
          - spanmetrics
      telemetry:
        logs:
          level: debug
        metrics:
          address: ${env:MY_POD_IP}:8888

Log output

Additional context

No response

[iblancasa]

I think the current behavior makes sense. You can aggregate them in your metrics backend.

[ixiaoyi93]

@iblancasa Sorry, I don't understand very much. Where should I summarize this data？

[iblancasa]

@iblancasa Sorry, I don't understand very much. Where should I summarize this data？

Each OpenTelemetry Collector instance runs its own metrics pipeline and emits runtime metrics like runtime_jvm_gc_timeOrCount_v2. These are emitted per collector process. If you're seeing multiple time series per node, that’s expected. I would recommend aggregating these metrics in your backend (e.g., using sum in Prometheus or equivalent) to view a node-level value.

[ixiaoyi93]

I have solved this problem, thank you.