[OTel to Prometheus] Promote resource attr `service.name`, `service.namespace`, and `service.instance.id` as Prometheus metric labels

[cyrille-leclerc]

What are you trying to achieve?

This proposal seeks to enhance the "OTLP Metric points to Prometheus" specification by introducing an option to expose the OpenTelemetry resource attributes service.name, service.namespace, and service.instance.id directly as Prometheus metric labels. These labels would appear on both the metrics themselves and the associated target_info metric.

Motivation

Currently, when converting OTLP metrics to Prometheus, these crucial service-identifying resource attributes are dropped in favor of the Prometheus job and instance$ labels. This omission creates several challenges for OpenTelemetry users:

• Intuitive Exploration: OpenTelemetry users typically think in terms of service.name and service.namespace when exploring their data. Forcing them to translate these concepts to Prometheus' job and instance can make metric exploration less intuitive and more difficult.

• Cross-Signal Correlation: Correlating Prometheus metrics with logs and traces becomes significantly harder. Most log and trace databases directly surface OpenTelemetry's service.name, service.namespace, and service.instance.id. Without these consistent labels on the Prometheus metrics, writing effective correlation queries is cumbersome.

Current Behavior Example

Consider the following OpenTelemetry metric:

OTEL
jvm.thread.count {
   resource {
      service.name=frontend, 
      service.namespace=webshop, 
      service.instance.id=inst-abcd
      service.version=1.2.3
   }
   attributes {
      jvm.thread.state=runnable
   }
} 2

Today, this converts to Prometheus as:

PROMETHEUS

# service_name, service_namespace, and service_instance_id labels
# are currently missing from the metric, hindering filtering on 
# OTel entities and correlation with logs and traces.

jvm_thread_duration {
   jvm_thread_state=runnable
   ...
   job=webshop/frontend
   instance=inst-abcd
}
target_info {
   service_version=1.2.3
   ...
   job=webshop/frontend
   instance=inst-abcd
}

Proposed Enhancement

We propose an option to promote service.name, service.namespace, and service.instance.id as Prometheus labels, resulting in metrics like this:

PROMETHEUS PROPOSAL

# With service_name, service_namespace, and service_instance_id labels
# available, querying by OpenTelemetry entities is simplified,
# and correlation with traces and logs is significantly improved.

jvm_thread_duration {
   jvm_thread_state=runnable
   ...
   service_namespace=webshop       # NEW
   service_name=frontend           # NEW
   service_instance_id=inst-abcd   # NEW
   job=webshop/frontend
   instance=inst-abcd
}
target_info {
   service_namespace=webshop      # NEW
   service_name=frontend.         # NEW
   service_instance_id=inst-abcd  # NEW
   service_version=1.2.3
   ...
   job=webshop/frontend
   instance=inst-abcd
}

Additional context.

It's worth noting that the Prometheus OTLP Endpoint offers related features:

• keep_identifying_resource_attributes: This promotes service.name, service.namespace, and service.instance.id to the target_info metric, but not to the metrics themselves.

• promote_resource_attributes: This allows promoting specified resource attributes as labels on the Prometheus metric. This feature is today commonly used to service.name, service.namespace, and service.instance.id alongside being used for infrastructure attributes like Kubernetes metadata. However, this feature doesn't solve the problem of promoting service.name|namespace|instance.id on target_info

This proposal aims to provide a more comprehensive solution for these widely used OpenTelemetry resource attributes.

_{Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[cyrille-leclerc]

Discrepancy in Prometheus Metric Exports: Prometheus OTLP Endpoint vs. OTel Java SDK Exporter for Prometheus

I've noticed an interesting difference in how Prometheus metrics are handled depending on the export method.

When using the OpenTelemetry Protocol (OTLP) endpoint for Prometheus, the service.name, service.namespace, and service.instance.id attributes are only included on target_info if keep_identifying_resource_attributes is explicitly set to true.

However, the Prometheus exporter within the OpenTelemetry Java SDK (specifically, open-telemetry/opentelemetry-java/tree/main/exporters/prometheus) appears to publish these same service attributes by default, regardless of any configuration.

This inconsistency could lead to varying data representations depending on the chosen export mechanism.

This confirms @dashpole hypothesis during the OpenTelemetry Prometheus Workgroup meeting on July 2nd.