Not able to receive metrics from logs count connector

[jayadeepdamarla]

Component(s)

connector/count

What happened?

Description

config as below, in mimir backend we are not able to get app.error.count metrics. I am not able to see count connector intialised and started in logs of collector version, 0.120.0.

Steps to Reproduce

Expected Result

Actual Result

Collector version

0.120.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

config:
  exporters:
    debug:
      verbosity: basic
    otlphttp:
      traces_endpoint: http://xxx/v1/traces
      logs_endpoint: http://xxx/otlp/v1/logs
      metrics_endpoint: http://xxx/otlp/v1/metrics
     
  extensions:
    # The health_check extension is mandatory for this chart.
    # Without the health_check extension the collector will fail the readiness and liveliness probes.
    # The health_check extension can be modified, but should never be removed.
    health_check:
      endpoint: ${env:MY_POD_IP}:13133
  processors:
    batch:
      timeout: 15s
      send_batch_size: 100
      send_batch_max_size: 1000
    # Default memory limiter configuration for the collector based on k8s resource limits.
    memory_limiter:
      # check_interval is the time between measurements of memory usage.
      check_interval: 5s
      # By default limit_mib is set to 80% of ".Values.resources.limits.memory"
      limit_percentage: 80
      # By default spike_limit_mib is set to 25% of ".Values.resources.limits.memory"
      spike_limit_percentage: 25
  connectors:
    spanmetrics:
      exemplars:
        enabled: true
    servicegraph:
    count:
      logs:
        app.error.count:
          description: "Error in application"
          # conditions:
          # - attributes["detected.level"] == "error"
          attributes:
          - key: "service.name"
            value: "service.name"
          - key: "RequestPath"
            value: "RequestPath"
          - key: "exception.type"
            value: "exception.type"
          - key: "detected.level"
            value: "detected.level"
    # sum:
    #   logs:
    #     app.error.count:
    #       source_attribute: detected_level
    #       conditions:
    #         - attributes["detected_level"] == "error"
    #       attributes:
    #       - key: "service_name"
    #         value: "service_name"
    #       - key: "RequestPath"
    #         value: "RequestPath"
    #       - key: "exception_type"
    #         value: "exception_type"

  receivers:
    # jaeger:
    #   protocols:
    #     grpc:
    #       endpoint: ${env:MY_POD_IP}:14250
    #     thrift_http:
    #       endpoint: ${env:MY_POD_IP}:14268
    #     thrift_compact:
    #       endpoint: ${env:MY_POD_IP}:6831
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          endpoint: ${env:MY_POD_IP}:4318
    # prometheus:
    #   config:
    #     scrape_configs:
    #       - job_name: opentelemetry-collector
    #         scrape_interval: 10s
    #         static_configs:
    #           - targets:
    #               - ${env:MY_POD_IP}:8888
    # zipkin:
    #   endpoint: ${env:MY_POD_IP}:9411
  service:
    telemetry:
      metrics:
        address: ${env:MY_POD_IP}:8888
    extensions:
    - health_check
    pipelines:
      logs:
        exporters:
        - otlphttp
        - debug
        - count
        processors:
        - memory_limiter
        - batch
        receivers:
        - otlp
      metrics:
        exporters:
        - otlphttp
        - debug
        processors:
        - memory_limiter
        - batch
        receivers:
        - otlp
        - prometheus
        - spanmetrics
        - servicegraph
        - count
      traces:
        exporters:
        - otlphttp
        - debug
        - spanmetrics
        - servicegraph
        processors:
        - memory_limiter
        - batch
        receivers:
        - otlp
        - jaeger
        - zipkin

Log output

stream logs failed container "opentelemetry-collector" in pod "otlp-collector-opentelemetry-collector-659446c9f-96vhs" is waiting to start: ContainerCreating for open-telemetry/otlp-collector-opentelemetry-collector-659446c9f-96vhs (opentelemetry-collector)
2025-03-03T14:44:07.966Z    info    service@v0.120.0/service.go:193    Setting up own telemetry...
2025-03-03T14:44:07.966Z    warn    service@v0.120.0/service.go:241    service::telemetry::metrics::address is being deprecated in favor of service::telemetry::metrics::readers
2025-03-03T14:44:07.967Z    info    builders/builders.go:26    Development component. May change in the future.    {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"}
2025-03-03T14:44:07.967Z    info    memorylimiter@v0.120.0/memorylimiter.go:151    Using percentage memory limiter    {"otelcol.component.kind": "Processor", "total_memory_mib": 512, "limit_percentage": 80, "spike_limit_percentage": 25}
2025-03-03T14:44:07.967Z    info    memorylimiter@v0.120.0/memorylimiter.go:75    Memory limiter configured    {"otelcol.component.kind": "Processor", "limit_mib": 409, "spike_limit_mib": 128, "check_interval": 5}
2025-03-03T14:44:07.967Z    info    spanmetricsconnector@v0.120.1/connector.go:110    Building spanmetrics connector    {"otelcol.component.id": "spanmetrics", "otelcol.component.kind": "Connector", "otelcol.signal": "traces", "otelcol.signal.output": "metrics"}
2025-03-03T14:44:07.967Z    info    builders/builders.go:26    Development component. May change in the future.    {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "logs"}
2025-03-03T14:44:07.967Z    info    builders/builders.go:26    Development component. May change in the future.    {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}
2025-03-03T14:44:08.070Z    info    service@v0.120.0/service.go:258    Starting otelcol-contrib...    {"Version": "0.120.1", "NumCPU": 4}
2025-03-03T14:44:08.070Z    info    extensions/extensions.go:40    Starting extensions...
2025-03-03T14:44:08.070Z    info    extensions/extensions.go:44    Extension is starting...    {"otelcol.component.id": "health_check", "otelcol.component.kind": "Extension"}
2025-03-03T14:44:08.070Z    info    healthcheckextension@v0.120.1/healthcheckextension.go:32    Starting health_check extension    {"otelcol.component.id": "health_check", "otelcol.component.kind": "Extension", "config": {"Endpoint":"10.2.32.169:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"ReadTimeout":0,"ReadHeaderTimeout":0,"WriteTimeout":0,"IdleTimeout":0,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2025-03-03T14:44:08.071Z    info    extensions/extensions.go:61    Extension started.    {"otelcol.component.id": "health_check", "otelcol.component.kind": "Extension"}
2025-03-03T14:44:08.071Z    info    spanmetricsconnector@v0.120.1/connector.go:204    Starting spanmetrics connector    {"otelcol.component.id": "spanmetrics", "otelcol.component.kind": "Connector", "otelcol.signal": "traces", "otelcol.signal.output": "metrics"}
2025-03-03T14:44:08.071Z    info    servicegraphconnector@v0.120.1/connector.go:155    Started servicegraphconnector    {"otelcol.component.id": "servicegraph", "otelcol.component.kind": "Connector", "otelcol.signal": "traces", "otelcol.signal.output": "metrics"}
2025-03-03T14:44:08.071Z    info    otlpreceiver@v0.120.0/otlp.go:116    Starting GRPC server    {"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "10.2.32.169:4317"}
2025-03-03T14:44:08.071Z    info    otlpreceiver@v0.120.0/otlp.go:173    Starting HTTP server    {"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "10.2.32.169:4318"}
2025-03-03T14:44:08.071Z    info    jaegerreceiver@v0.120.1/trace_receiver.go:241    Starting UDP server for Compact Thrift    {"otelcol.component.id": "jaeger", "otelcol.component.kind": "Receiver", "otelcol.signal": "traces", "endpoint": "10.2.32.169:6831"}
2025-03-03T14:44:08.071Z    info    jaegerreceiver@v0.120.1/trace_receiver.go:347    Starting HTTP server for Jaeger Thrift    {"otelcol.component.id": "jaeger", "otelcol.component.kind": "Receiver", "otelcol.signal": "traces", "endpoint": "10.2.32.169:14268"}
2025-03-03T14:44:08.071Z    info    jaegerreceiver@v0.120.1/trace_receiver.go:372    Starting gRPC server for Jaeger Protobuf    {"otelcol.component.id": "jaeger", "otelcol.component.kind": "Receiver", "otelcol.signal": "traces", "endpoint": "10.2.32.169:14250"}
2025-03-03T14:44:08.073Z    info    prometheusreceiver@v0.120.1/metrics_receiver.go:119    Starting discovery manager    {"otelcol.component.id": "prometheus", "otelcol.component.kind": "Receiver", "otelcol.signal": "metrics"}
2025-03-03T14:44:08.073Z    info    targetallocator/manager.go:184    Scrape job added    {"otelcol.component.id": "prometheus", "otelcol.component.kind": "Receiver", "otelcol.signal": "metrics", "jobName": "opentelemetry-collector"}
2025-03-03T14:44:08.073Z    info    healthcheck/handler.go:132    Health Check state change    {"otelcol.component.id": "health_check", "otelcol.component.kind": "Extension", "status": "ready"}
2025-03-03T14:44:08.073Z    info    service@v0.120.0/service.go:281    Everything is ready. Begin running and processing data.
2025-03-03T14:44:08.073Z    info    prometheusreceiver@v0.120.1/metrics_receiver.go:188    Starting scrape manager    {"otelcol.component.id": "prometheus", "otelcol.component.kind": "Receiver", "otelcol.signal": "metrics"}

Additional context

No response

[github-actions]

Pinging code owners:

• connector/count: @djaglowski @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bacherfl]

Hi @jayadeepdamarla can you please verify that the log entries sent to the collector contain the required attributes specified in the count connector? I.e. those:

attributes:
          - key: "service.name"
            value: "service.name"
          - key: "RequestPath"
            value: "RequestPath"
          - key: "exception.type"
            value: "exception.type"
          - key: "detected.level"
            value: "detected.level"

If any of these is missing, and no default value is configured, the received entry will not contribute to the count, so maybe that is why the metric is not generated in this case.