Label Dropping Issue in OTel Prometheus Receiver for aerospike_vector_search_requests

[TechHunter07]

Component(s)

No response

What happened?

Note:

For the metric aerospike_vector_search_requests, I have attached screenshots from both the Prometheus endpoint (showing the full set of labels and multiple series) and the OpenTelemetry Collector output (showing missing labels and dropped series). Only for this metric only it dropping the labels other metrics where as single dataend

Screenshot 1: Raw Prometheus endpoint output
(Screenshot showing aerospike_vector_search_requests with multiple series, including the status label)

Screenshot 2: OpenTelemetry Collector processed output
(Screenshot showing aerospike_vector_search_requests with missing labels and reduced series count)

Screenshot 3: Comparison metric aerospike_vector_search_internal_connections, which retains the status label correctly.
This highlights that only aerospike_vector_search_requests is experiencing label loss, while other metrics (like aerospike_vector_search_internal_connections) retain their labels properly.

Description

When the OTEL Collector’s Prometheus receiver scrapes a Prometheus-formatted endpoint, the raw output contains multiple time series for summary metrics (e.g., aerospike_vector_search_requests) with a full set of labels—including the status label on some series. However, after processing, the collector exports only one aggregated series, and the status label (and others associated with zero-value series) is missing for the above metric only. It appears that the collector’s internal aggregation and zero-value filtering logic is causing these series—and their labels—to be dropped, even though honor_labels: true is set.

Steps to Reproduce

Configure the OTEL Collector with a Prometheus receiver that scrapes an endpoint exposing summary metrics with all labels for a metric

Expected Result

All time series as scraped by Prometheus should be preserved—including all label combinations (such as the status label) and zero values—so that the final exported metrics exactly match the raw endpoint data.

Actual Result

Actual Result

Only one series is exported:

aerospike_vector_search_requests{cluster_name="prism-image-search",instance="prism-image-search_avs_1:5040",job="vector",namespace="avs-data",type="write"} 4
This output is missing several series (and their associated labels, such as status) visible in the raw Prometheus endpoint.

Collector version

otel/opentelemetry-collector-contrib:latest

Environment information

OS: (e.g., "Ubuntu 20.04")
Running the entire setup in Docker

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "as_otel_contrib:4317"

  prometheus:
    config:
      scrape_configs:
        - job_name: 'vector'
          scrape_interval: 10s
          honor_labels: true
          static_configs:
            - targets: ['prism-image-search_avs_1:5040']
          metrics_path: /manage/rest/v1/prometheus

exporters:
  prometheus:
    endpoint: "0.0.0.0:30145"

  datadog:
    api: 
      site: ######################
      key: ###############
    metrics:
      resource_attributes_as_tags: true
      sums:
        cumulative_monotonic_mode: raw_value

processors:
  # First processor: Rename metrics and update labels using the metricstransform processor.
  metricstransform:
    transforms:
      - include: ^aerospike_(.*)$$
        match_type: regexp
        action: update
        new_name: aerospike.aerospike_$$1
      - include: ^aerospike(.*)$$
        match_type: regexp
        action: update
        operations:
          - action: update_label
            label: service
            new_label: aerospike_service
          - action: update_label
            label: cluster_name
            new_label: aerospike_cluster

  # Second processor: Use the transform processor (OTTL) to copy the resource attribute.
  transform:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["secret_instance"], resource.attributes["service.instance.id"]) where IsMatch(metric.name, "^aerospike\\.aerospike_sa_.*$")

  batch:
        send_batch_max_size: 100
        send_batch_size: 10
        timeout: 15s

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog ]
    metrics:
      receivers: [prometheus ]
      processors: [metricstransform, transform, batch]
      exporters: [prometheus, datadog]

Log output

## Prometheus Endpoint result:
curl -s http://prism-image-search_avs_1:5040/manage/rest/v1/prometheus | grep "aerospike_vector_search_requests"

# HELP aerospike_vector_search_requests Generated from Dropwizard metric import (metric=requests-malformed, type=com.aerospike.connect.metrics.AerospikeCounter)
# TYPE aerospike_vector_search_requests gauge
aerospike_vector_search_requests{status="malformed",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="unauthenticated-error",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="error",type="delete",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="delete",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="index-queue-full-error",type="update",update_type="client",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="index-queue-full-ignore",type="update",update_type="client",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="error",type="is-indexed",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="is-indexed",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 56.0
aerospike_vector_search_requests{status="error",type="query",query_type="client",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="error",type="query",query_type="local",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="queued",type="query",query_type="local",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="query",query_type="local",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 3.0
aerospike_vector_search_requests{status="error",type="query",query_type="proxy-in",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="queued",type="query",query_type="proxy-in",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="query",query_type="proxy-in",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="error",type="query",query_type="proxy-out",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="query",query_type="proxy-out",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="query",query_type="client",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 3.0
aerospike_vector_search_requests{status="error",type="update",update_type="client",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="error",type="update",update_type="local",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="queued",type="update",update_type="local",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="update",update_type="local",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 4.0
aerospike_vector_search_requests{status="error",type="update",update_type="proxy-in",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="queued",type="update",update_type="proxy-in",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="update",update_type="proxy-in",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="error",type="update",update_type="proxy-out",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="update",update_type="proxy-out",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="update",update_type="client",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 4.0
aerospike_vector_search_requests{status="error",type="delete",namespace="avs-data",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="delete",namespace="avs-data",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="error",type="exists",namespace="avs-data",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="exists",namespace="avs-data",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="error",type="read",namespace="avs-data",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="read",namespace="avs-data",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="error",type="write",namespace="avs-data",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{status="success",type="write",namespace="avs-data",cluster_name="prism-image-search",} 4.0
aerospike_vector_search_requests{quantile="0.5",type="delete",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests{quantile="0.75",type="delete",index="avs-data-prism-image-search",cluster_name="prism-image-search",} 0.0
aerospike_vector_search_requests

##Otel Collector output:
curl -s http://as_otel_contrib:30145/metrics | grep "aerospike_vector_search_requests"

# HELP aerospike_aerospike_vector_search_requests Generated from Dropwizard metric import (metric=requests-malformed, type=com.aerospike.connect.metrics.AerospikeCounter)
# TYPE aerospike_aerospike_vector_search_requests gauge
aerospike_aerospike_vector_search_requests{aerospike_cluster="prism-image-search",instance="prism-image-search_avs_1:5040",job="vector",namespace="avs-data",type="write"} 4

Additional context

Additional context:
It appears that the OTEL Collector’s Prometheus receiver is aggregating or filtering out zero-value series (and associated labels) during conversion to OTLP metrics. While Prometheus’s native scraping shows every series (even those with zero values), the Collector is optimized to drop redundant zero-value data, which in this case results in the loss of the “status” label in quantile series for summary metrics. This behavior is problematic because downstream systems (e.g., Datadog) receive incomplete label sets.

I’m raising this issue to seek guidance or a fix that would allow preserving all label combinations—including those on zero-value series—so that the exported metrics mirror exactly what is scraped from the endpoint.

[github-actions]

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[ArthurSens]

Since one of your metrics has the status label, I don't think there's a problem in the prometheus receiver.

You could try removing the processors and re-adding them one by one until you figure out which one is removing the label you wanted to keep.

[VENKATA-SIVA-OPTIT]

Hi @ArthurSens I tried removing processors individually, but I cannot collect all metric values or numbers along with status label.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheusreceiver: @Aneurysm9 @dashpole @ArthurSens @krajorama

See Adding Labels via Comments if you do not have permissions to add labels yourself.