[exporter/prometheus] Exemplars not exposed in Prometheus /metrics despite being received by OpenTelemetry Collector

[dheeraj-vanamala]

Component(s)

exporter/prometheus

What happened?

Description

When using the exporter/prometheus in OpenTelemetry Collector, exemplars are successfully generated by a manually instrumented Python application and received by the OTEL Collector. These exemplars are visible in the OTEL Collector debug logs, confirming that the trace-metric correlation works internally.

However, no exemplars are being exposed via the /metrics endpoint scraped by Prometheus, even though all configurations are set correctly and trace IDs are present in the metric logs.

Steps to Reproduce

1. Start the stack using the following files (all uploaded in this issue):
   • docker-compose.yml (Prometheus, Grafana)
   • docker-compose.otel.yml (OpenTelemetry Collector, Jaeger)
   • otel-collector-config.yml (OpenTelemetry Collector configuration)
   • prometheus-config.yml (Prometheus configuration)
2. Run the instrumented Python app with the command:

OTEL_EXEMPLARS_SAMPLING_PROBABILITY=1.0 OTEL_EXEMPLAR_FILTER=always_on python server_a.py

3. While the app is running, generate traffic with the command:

for i in {1..6}; do curl http://localhost:6000/test; sleep 1; done

4. Observe metrics in real-time:

• The Python server prints metrics with exemplars in the terminal when the /test endpoint is hit.
• Prometheus is scraping the /metrics endpoint from the OpenTelemetry Collector.
• We are not saving the data to any files — everything is visible in live terminal output.

Expected Result

• The /metrics endpoint exposed by OTEL Collector should contain exemplar annotations attached to the metric lines.
• Prometheus should successfully scrape and store metrics with trace exemplars for correlation.

Actual Result

• Exemplar annotations (e.g., # {trace_id="..."}) are missing entirely from the /metrics endpoint.
• Prometheus receives only raw metrics with no exemplar data, making trace-metric correlation impossible.

Collector version

v0.122.1

Environment information

Environment

OS: macOS Sonoma 14.3.1 (Apple Silicon - M2)
Docker: Docker Desktop v4.28.0 (Engine: 24.0.7, Compose: v2.24.5)
Python: 3.9.18
otel-collector image: otel/opentelemetry-collector-contrib:0.122.1
Prometheus: v3.2.1
Grafana: latest (as of March 2025)
Jaeger: jaegertracing/all-in-one:1.42.0

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"  # Accepts traces from both local and Docker
      http:
        endpoint: "0.0.0.0:4318"  # Accepts HTTP traces

processors:
  batch:

exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: "http://jaeger:4317"
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:9464"
    enable_open_metrics: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug, otlp]
      processors: [batch]

    metrics:  
      receivers: [otlp]
      exporters: [debug, prometheus]
      processors: [batch]

Log output

### 🔹 OTEL Collector Logs (with debug enabled)

➜  prometheus-stack docker logs -f otel-collector
2025-03-22T17:50:25.473Z	info	service@v0.122.1/service.go:193	Setting up own telemetry...
2025-03-22T17:50:25.473Z	info	builders/builders.go:26	Development component. May change in the future.	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"}
2025-03-22T17:50:25.478Z	info	builders/builders.go:26	Development component. May change in the future.	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}
2025-03-22T17:50:25.480Z	info	service@v0.122.1/service.go:260	Starting otelcol-contrib...	{"Version": "0.122.1", "NumCPU": 8}
2025-03-22T17:50:25.480Z	info	extensions/extensions.go:40	Starting extensions...
2025-03-22T17:50:25.481Z	info	otlpreceiver@v0.122.1/otlp.go:116	Starting GRPC server	{"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "0.0.0.0:4317"}
2025-03-22T17:50:25.481Z	info	otlpreceiver@v0.122.1/otlp.go:173	Starting HTTP server	{"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "0.0.0.0:4318"}
2025-03-22T17:50:25.481Z	info	service@v0.122.1/service.go:283	Everything is ready. Begin running and processing data.
2025-03-22T17:51:31.574Z	info	Metrics	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 2, "data points": 4}
2025-03-22T17:51:31.575Z	info	ResourceMetrics #0
Resource SchemaURL:
Resource attributes:
     -> service.name: Str(service-a)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope __main__
Metric #0
Descriptor:
     -> Name: custom_requests_total
     -> Description: Total custom requests
     -> Unit:
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61)
StartTimestamp: 2025-03-22 17:51:30.888351 +0000 UTC
Timestamp: 2025-03-22 17:51:31.55833 +0000 UTC
Value: 1
Exemplars:
Exemplar #0
     -> Trace ID: c19a57ed96ca1db94aeb773a15145851
     -> Span ID: a55aa0c81d7d9d85
     -> Timestamp: 2025-03-22 17:51:30.888233 +0000 UTC
     -> Value: 1
NumberDataPoints #1
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65)
StartTimestamp: 2025-03-22 17:51:31.440623 +0000 UTC
Timestamp: 2025-03-22 17:51:31.55833 +0000 UTC
Value: 1
Exemplars:
Exemplar #0
     -> Trace ID: 2f7f0222c12884db24145c107a537840
     -> Span ID: 798c6d5f7eff0460
     -> Timestamp: 2025-03-22 17:51:31.440423 +0000 UTC
     -> Value: 1
Metric #1
Descriptor:
     -> Name: custom_request_duration
     -> Description: Custom request duration
     -> Unit: ms
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61)
StartTimestamp: 2025-03-22 17:51:30.900367 +0000 UTC
Timestamp: 2025-03-22 17:51:31.55833 +0000 UTC
Count: 1
Sum: 12.521982
Min: 12.521982
Max: 12.521982
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
Exemplars:
Exemplar #0
     -> Trace ID: c19a57ed96ca1db94aeb773a15145851
     -> Span ID: a55aa0c81d7d9d85
     -> Timestamp: 2025-03-22 17:51:30.900251 +0000 UTC
     -> Value: 12.521982
HistogramDataPoints #1
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65)
StartTimestamp: 2025-03-22 17:51:31.440736 +0000 UTC
Timestamp: 2025-03-22 17:51:31.55833 +0000 UTC
Count: 1
Sum: 12.529135
Min: 12.529135
Max: 12.529135
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
Exemplars:
Exemplar #0
     -> Trace ID: 2f7f0222c12884db24145c107a537840
     -> Span ID: 798c6d5f7eff0460
     -> Timestamp: 2025-03-22 17:51:31.440671 +0000 UTC
     -> Value: 12.529135
	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"}
2025-03-22T17:51:32.589Z	info	Metrics	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 2, "data points": 4}
2025-03-22T17:51:32.589Z	info	ResourceMetrics #0
Resource SchemaURL:
Resource attributes:
     -> service.name: Str(service-a)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope __main__
Metric #0
Descriptor:
     -> Name: custom_requests_total
     -> Description: Total custom requests
     -> Unit:
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61)
StartTimestamp: 2025-03-22 17:51:30.888351 +0000 UTC
Timestamp: 2025-03-22 17:51:32.571423 +0000 UTC
Value: 1
NumberDataPoints #1
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65)
StartTimestamp: 2025-03-22 17:51:31.440623 +0000 UTC
Timestamp: 2025-03-22 17:51:32.571423 +0000 UTC
Value: 1
Metric #1
Descriptor:
     -> Name: custom_request_duration
     -> Description: Custom request duration
     -> Unit: ms
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61)
StartTimestamp: 2025-03-22 17:51:30.900367 +0000 UTC
Timestamp: 2025-03-22 17:51:32.571423 +0000 UTC
Count: 1
Sum: 12.521982
Min: 12.521982
Max: 12.521982
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
HistogramDataPoints #1
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65)
StartTimestamp: 2025-03-22 17:51:31.440736 +0000 UTC
Timestamp: 2025-03-22 17:51:32.571423 +0000 UTC
Count: 1
Sum: 12.529135
Min: 12.529135
Max: 12.529135
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"}
2025-03-22T17:51:33.601Z	info	Metrics	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 2, "data points": 4}
2025-03-22T17:51:33.601Z	info	Traces	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 2}
2025-03-22T17:51:33.602Z	info	ResourceMetrics #0
Resource SchemaURL:
Resource attributes:
     -> service.name: Str(service-a)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope __main__
Metric #0
Descriptor:
     -> Name: custom_requests_total
     -> Description: Total custom requests
     -> Unit:
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61)
StartTimestamp: 2025-03-22 17:51:30.888351 +0000 UTC
Timestamp: 2025-03-22 17:51:33.580137 +0000 UTC
Value: 1
NumberDataPoints #1
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65)
StartTimestamp: 2025-03-22 17:51:31.440623 +0000 UTC
Timestamp: 2025-03-22 17:51:33.580137 +0000 UTC
Value: 1
Metric #1
Descriptor:
     -> Name: custom_request_duration
     -> Description: Custom request duration
     -> Unit: ms
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61)
StartTimestamp: 2025-03-22 17:51:30.900367 +0000 UTC
Timestamp: 2025-03-22 17:51:33.580137 +0000 UTC
Count: 1
Sum: 12.521982
Min: 12.521982
Max: 12.521982
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
HistogramDataPoints #1
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65)
StartTimestamp: 2025-03-22 17:51:31.440736 +0000 UTC
Timestamp: 2025-03-22 17:51:33.580137 +0000 UTC
Count: 1
Sum: 12.529135
Min: 12.529135
Max: 12.529135
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"}
2025-03-22T17:51:33.602Z	info	ResourceSpans #0
Resource SchemaURL:
Resource attributes:
     -> service.name: Str(service-a)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope __main__
Span #0
    Trace ID       : c19a57ed96ca1db94aeb773a15145851
    Parent ID      :
    ID             : a55aa0c81d7d9d85
    Name           : custom-operation
    Kind           : Internal
    Start time     : 2025-03-22 17:51:30.875631 +0000 UTC
    End time       : 2025-03-22 17:51:30.900463 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> http.method: Str(GET)
Span #1
    Trace ID       : 2f7f0222c12884db24145c107a537840
    Parent ID      :
    ID             : 798c6d5f7eff0460
    Name           : custom-operation
    Kind           : Internal
    Start time     : 2025-03-22 17:51:31.4278 +0000 UTC
    End time       : 2025-03-22 17:51:31.440866 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> http.method: Str(GET)
	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}
2025-03-22T17:51:34.011Z	info	Metrics	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 2, "data points": 4}
2025-03-22T17:51:34.011Z	info	ResourceMetrics #0
Resource SchemaURL:
Resource attributes:
     -> service.name: Str(service-a)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope __main__
Metric #0
Descriptor:
     -> Name: custom_requests_total
     -> Description: Total custom requests
     -> Unit:
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61)
StartTimestamp: 2025-03-22 17:51:30.888351 +0000 UTC
Timestamp: 2025-03-22 17:51:33.855355 +0000 UTC
Value: 1
NumberDataPoints #1
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65)
StartTimestamp: 2025-03-22 17:51:31.440623 +0000 UTC
Timestamp: 2025-03-22 17:51:33.855355 +0000 UTC
Value: 1
Metric #1
Descriptor:
     -> Name: custom_request_duration
     -> Description: Custom request duration
     -> Unit: ms
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61)
StartTimestamp: 2025-03-22 17:51:30.900367 +0000 UTC
Timestamp: 2025-03-22 17:51:33.855355 +0000 UTC
Count: 1
Sum: 12.521982
Min: 12.521982
Max: 12.521982
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
HistogramDataPoints #1
Data point attributes:
     -> http.method: Str(GET)
     -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65)
StartTimestamp: 2025-03-22 17:51:31.440736 +0000 UTC
Timestamp: 2025-03-22 17:51:33.855355 +0000 UTC
Count: 1
Sum: 12.529135
Min: 12.529135
Max: 12.529135
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"}

---

### 🔹 Prometheus /metrics API Output (filtered)

➜  ~ curl http://localhost:9464/metrics
# HELP custom_request_duration_milliseconds Custom request duration
# TYPE custom_request_duration_milliseconds histogram
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="0"} 0
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="5"} 0
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="10"} 0
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="25"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="50"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="75"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="100"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="250"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="500"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="750"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="1000"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="2500"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="5000"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="7500"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="10000"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="+Inf"} 1
custom_request_duration_milliseconds_sum{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61"} 12.521982192993164
custom_request_duration_milliseconds_count{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="0"} 0
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="5"} 0
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="10"} 0
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="25"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="50"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="75"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="100"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="250"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="500"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="750"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="1000"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="2500"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="5000"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="7500"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="10000"} 1
custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="+Inf"} 1
custom_request_duration_milliseconds_sum{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65"} 12.529134750366211
custom_request_duration_milliseconds_count{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65"} 1
# HELP custom_requests_total Total custom requests
# TYPE custom_requests_total counter
custom_requests_total{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61"} 1
custom_requests_total{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65"} 1

Additional context

Additional Context

• Tried multiple versions of Prometheus (v2.50.0, v3.2.1) and OpenTelemetry Collector (v0.122.1).
• Verified that the exemplar-related OTEL environment variables were correctly set:
  • OTEL_EXEMPLARS_SAMPLING_PROBABILITY=1.0
  • OTEL_EXEMPLAR_FILTER=always_on
• Metrics with exemplars are confirmed to be emitted in OTEL collector logs while the request is in flight.
• Prometheus /metrics scrape does not contain any exemplar lines even during active request traffic.
• Prometheus is running with --enable-feature=exemplar-storage and --enable-feature=otlp-write-receiver.
• Exemplar fields are printed in OTEL logs, but they don’t make it to Prometheus scrape endpoint.
• Correlation attempt was done using manual instrumentation with OpenTelemetry SDK (Python).
• Used two custom metrics:
  • Counter: custom_requests_total
  • Histogram: custom_request_duration
• Both metrics include the following attributes:
  • http.method (example: "GET")
  • request_id (example: uuid.uuid4() to avoid exemplar overwriting)
• Ensured that:
  • Prometheus scrape interval is set to 1s
  • OTEL export interval is set to 1s
• Auto-instrumentation failed to emit exemplars previously; switched to manual instrumentation for clarity.
• Attempted to use enable_open_metrics: true and enable_exemplars: true in Prometheus exporter config. However, the collector failed to start when enable_exemplars: true was added.

🐍 Python Server Instrumentation Summary

The Flask app is manually instrumented for both traces and metrics using the OpenTelemetry SDK. Custom metrics and exemplars are emitted using the /test endpoint.

A snippet of the relevant /test route logic that emits exemplars:

• Histogram: custom_request_duration
• Counter: custom_requests_total
• request_id attribute is added to both metrics and spans

Tracer and Meter setup with:

• OTLPSpanExporter (to Jaeger)
• OTLPMetricExporter (to Collector)
• ConsoleMetricExporter for debug output

📦 Attached Archive: exemplar-correlation-setup.zip

This archive includes all relevant files used to reproduce the issue:

• server_a.py: Manually instrumented Python app with traces + metrics
• otel-collector-config.yml: OTEL config with Prometheus + Jaeger exporters
• docker-compose.yml: Prometheus, Grafana, Node Exporter
• docker-compose.otel.yml: OTEL Collector and Jaeger
• prometheus.yml: Scrape configuration with 1s interval

exemplar-correlation-setup.zip

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole @ArthurSens

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dheeraj-vanamala]

🔁 Update: We were finally able to get exemplars to show up in the /metrics endpoint after a lot of trial and error.

We discovered that:

• Exemplars only appear while a request is actively in-flight, and are very time-sensitive.
• They are emitted only when /metrics is scraped during the request, and with the correct header:

curl -H “Accept: application/openmetrics-text” http://localhost:9464/metrics

• Once the request completes, exemplar annotations disappear from the metrics output.

---

💡 Request for Maintainer Insights

We’d love to understand how things work under the hood:

1. What’s the internal mechanism that causes exemplars to appear only during in-flight requests?
2. Why do they disappear immediately after the request ends?
3. Is this expected behavior or a side-effect of the current exemplar storage strategy in the Collector → Prometheus flow?
4. Can this be stabilized? Especially for auto-instrumentation setups where we don’t want to rewrite code for manual spans.
5. Are there any plans to improve this behavior or enhance observability in Collector configs?

---

🧪 Auto-Instrumentation Clarity

We also struggled with auto-instrumentation — it did not emit exemplars at all.
Would love guidance on:

• Whether auto-instrumentation + exemplars are supported
• Any recommended patterns to make it work reliably

---

Thanks in advance 🙏 and appreciate all the hard work you put into this ecosystem.

[ArthurSens]

Pinging @jmichalek132, as he wants to investigate this further

[jmichalek132]

I will take a deeper look later this week, but even before that some things I would suggest trying is using the OTLP receiver in prometheus to ingest OTLP directly from app / or collector or using the remote write exporter from the collector to prometheus instance to see if you will have the same issue..