Description
Component(s)
exporter/kafka
What happened?
Description
With the default configuration, the Kafka exporter fails to produce data. Explicitly configuring topic: otlp_metrics fixes this -- but that is deprecated configuration.
Steps to Reproduce
Run Redpanda locally (or Kafka if you like): docker run --publish=9093:9093 --health-cmd "rpk cluster health | grep 'Healthy:.*true'" docker.redpanda.com/redpandadata/redpanda:v23.1.11 redpanda start --kafka-addr=internal://0.0.0.0:9092,external://0.0.0.0:9093 --smp=1 --memory=1G --mode=dev-container
Run the collector with the config below.
Expected Result
Kafka broker metrics should be scraped, exported to Kafka, received back from Kafka, and then exported to the debug exporter (terminal).
Actual Result
The collector runs, but logs errors in a loop. Notably this:
2025-05-02T10:55:20.494+0800 info internal/retry_sender.go:133 Exporting failed. Will retry the request after interval. {"otelcol.component.id": "kafka", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "error": "Failed to deliver 1 messages due to kafka server: The request attempted to perform an operation on an invalid topic", "interval": "5.643311441s"}
Collector version
v0.125.0
Environment information
Environment
OS: Ubuntu 24.04
Compiler(if manually compiled): go1.24.1
OpenTelemetry Collector configuration
receivers:
kafka:
brokers: [localhost:9093]
kafkametrics:
brokers: [localhost:9093]
collection_interval: 10s
scrapers: [topics, consumers]
exporters:
debug:
verbosity: detailed
kafka:
brokers: [localhost:9093]
service:
pipelines:
metrics/kafka:
receivers: [kafka]
exporters: [debug]
metrics:
receivers: [kafkametrics]
exporters: [kafka]Log output
2025-05-02T10:55:20.494+0800 info internal/retry_sender.go:133 Exporting failed. Will retry the request after interval. {"otelcol.component.id": "kafka", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "error": "Failed to deliver 1 messages due to kafka server: The request attempted to perform an operation on an invalid topic", "interval": "5.643311441s"}Additional context
No response