[exporter/prometheusremotewrite] Add Detailed Export Failure Metrics

[mt-hasan]

Component(s)

exporter/prometheusremotewrite

Is your feature request related to a problem? Please describe.

The prometheusremotewrite exporter currently lacks detailed metrics and logs for export failures. When issues like timeouts or authorization errors occur, users often encounter generic error messages, making it challenging to diagnose and address the root causes effectively.

Problem:

1. No clear metrics when queue isn't full but sends fail
2. Limited visibility into failure types

{"level":
    "error",
    "timestamp":"2025-04-25T19:38:22.027Z",
    "caller":"exporterhelper/queue_sender.go:90",
    "message":"Exporting failed. Dropping data.",
    "kind":"exporter",
    "data_type":"metrics",
    "name":"prometheusremotewrite",
    "error":"Permanent error: Permanent error: context deadline exceeded",
    "dropped_items":5,
    "stack":"go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender ...
}

Describe the solution you'd like

We propose enhancing the PRW exporter to provide more granular metrics and logs for export failures.

• Granular Failure Metrics:
  • Introduce a metric prw_export_failures_total with a reason label to categorize failure types:
    • HTTP status code families (4xx, 5xx)
    • Specific error types (e.g., "out of order sample", "timeout", "authorization")
    • Add prw_export_retries_total to count retry attempts.
  • Improved Logging:
    • Structured error messages with clear failure categorization
    • Include relevant debugging details (status codes, error messages)

These enhancements would significantly improve observability and help users quickly identify and resolve issues in their data export pipeline, regardless of the specific remote write endpoint they're using (e.g., Prometheus, Cortex, Thanos, or cloud-based solutions)

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil @dashpole @ArthurSens

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[rapphil]

Still on the missing metrics, would be nice to have a metric for number of samples & time series ingested into the remote prometheus. I'm suggesting this since there is no easy 1:1 relationship between OTEL metrics and prometheus metrics, specially in the case of classic histograms.

[jmichalek132]

Still on the missing metrics, would be nice to have a metric for number of samples & time series ingested into the remote prometheus. I'm suggesting this since there is no easy 1:1 relationship between OTEL metrics and prometheus metrics, specially in the case of classic histograms.

Yeah this might be also bit easier to implement accurately especially with PRW2 with the new required response headers https://prometheus.io/docs/specs/prw/remote_write_spec_2_0/#required-written-response-headers

[ArthurSens]

Hey hey, agreed that we're lacking metrics. We documented this in #39706, but we haven't gotten into the details of what exactly is missing. I really really want to promote our component to Stable, but I need some extra hands 😬

Would you be willing to help us here? 🙂

What we have listed is:

• Review component's observability. (This task will create new issues.)
  • Review throughput metrics
  • Review data drop metrics
  • Review correctness metrics
  • Review performance metrics/spans
  • Review error handling logs

So the ways people can help are:

• Document what kind of data goes in and out of the component, and what we need to measure to ensure we can monitor throughput.
• Document when we drop data for various reasons, such as OTel->Prometheus incompatibility, internal errors, retry failures, etc., and how to measure it.
• Document when we output something that is not totally correct, if we ever do. It's a bit fuzzy, but we may be intentionally not compliant with the OTel spec, and people need to know when this is happening.
• Document what we care about regarding performance and how to measure it.
• Document what internal errors our users should be aware of.

Of course, this doesn't need to be work for a single person. We can take baby steps, document only one of those, and implement. Then we move to the next one