Not able to modify the instance label in prometheus receiver

[spy16]

Component(s)

receiver/prometheus

What happened?

Description

I am using otelcol-contrib in agent mode (i.e., running on the same host as the application).

I have my Go app pushing metrics to otelcol-contrib via OTLP. Then, i am using prometheus receiver to scrape the node-exporter running on the same EC2 instance. Then using prometheus exporter to expose the combined metrics.

Both metrics are being emitted. But the node-exporter metrics have instance=localhost:9100 from all nodes. I want to change this to the hostname. So I tried the following setup to override the instance label using transform processor. But this doesn't change and causes the duplicate label error reported below.

Steps to Reproduce

Expected Result

I expected the label instance to be overriden with the value of host.name

Actual Result

Collector version

otelcol-contrib version 0.104.0

Environment information

Environment

Linux ip-50-0-50-55 6.8.0-1010-aws #10-Ubuntu SMP Thu Jun 13 17:48:42 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  prometheus:
    config:
      scrape_configs:
      - job_name: 'system-collector'
        scrape_interval: 1s
        static_configs:
          - targets: ['localhost:9100']

processors:
  batch:
  transform:
    metric_statements:
      - context: datapoint
        statements:
        - set(attributes["instance"], resource.attributes["host.name"])
  resourcedetection:
    detectors: [env, system]
    timeout: 2s
    override: false
    system:
      hostname_sources: ["dns", "os"]

exporters:
  debug:
  prometheus:
    endpoint: 0.0.0.0:9888

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

    metrics:
      receivers: [otlp, prometheus]
      processors: [batch, resourcedetection, transform]
      exporters: [debug, prometheus]

Log output

Jul 18 05:21:06 ip-50-0-50-55 otelcol-contrib[5723]: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
Jul 18 05:21:06 ip-50-0-50-55 otelcol-contrib[5723]:         github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/collector.go:382
Jul 18 05:21:06 ip-50-0-50-55 otelcol-contrib[5723]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
Jul 18 05:21:06 ip-50-0-50-55 otelcol-contrib[5723]:         github.com/prometheus/[email protected]/prometheus/registry.go:457
Jul 18 05:21:06 ip-50-0-50-55 otelcol-contrib[5723]: 2024-07-18T05:21:05.991Z        error        [email protected]/collector.go:382        failed to convert metric node_disk_discards_merged_total: duplicate label names in constant and variable labels for metric "node_disk_discards_merged_total"        {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
Jul 18 05:21:06 ip-50-0-50-55 otelcol-contrib[5723]: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
Jul 18 05:21:06 ip-50-0-50-55 otelcol-contrib[5723]:         github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/collector.go:382
Jul 18 05:21:06 ip-50-0-50-55 otelcol-contrib[5723]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
Jul 18 05:21:06 ip-50-0-50-55 otelcol-contrib[5723]:         github.com/prometheus/[email protected]/prometheus/registry.go:457

Additional context

I know the pipeline overall works as I am expectign since if i change the set(attributes["instance"]... to set(attributes["foo"],....) , the foo label appears with the expected values.

I think the instance label is being added similarly. but when prometheus exporter tries to format it for scraping, it is erroring out since it finds two values of instance

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

You have added an "instance" label to each metric's datapoints, rather than to the resource.

You should be able to do:

  transform:
    metric_statements:
      - context: resource
        statements:
        - set(attributes["instance"],attributes["host.name"])

[github-actions]

Pinging code owners for processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[spy16]

@dashpole I changed my transform to this:

  transform:
    metric_statements:
      - context: resource
        statements:
        - set(attributes["instance"], attributes["host.name"])

And then used the following exporter:

exporters:
  prometheus:
    endpoint: 0.0.0.0:9898
    resource_to_telemetry_conversion:
      enabled: false

Without telemetry conversion (i.e., enabled: false), the instance label does not get overriden at all. I still see localhost:9100 when i do curl localhost:9898/metrics

If I enable it, I again get the same conflict error you see above.

I tried adding a delete_key(attributes, "instance") as well in the transform (before set()). Even then the original instance=localhost:9100 is not going away.

P.S. Please suggest if there is any easier alternative to this. All I want is to add the host-name as instance label for all metrics from this otel-collector and send it to prometheus (via remote write). So that I can use the Node Exporter dashbaord from prometheus gallery.

[dashpole]

Ah, I forgot. Can you do:

  transform:
    metric_statements:
      - context: resource
        statements:
        - set(attributes["service.instance.id"], attributes["host.name"])

The prom exporter uses service.instance.id as instance

[spy16]

Oh yess! Thank you so much! that works as expected.

[spy16]

Closing this since this clearly is not a bug.