Issues with metricrelabeling for podmonitor

[laxmansoma22]

Component(s)

No response

What happened?

Description

I am trying to keep certain metrics for kube-state-metrics , we started using otel collector target allocator recently instead of kube prometheus stack as its working with same template...

Steps to Reproduce

metricRelabelings:
- action: labeldrop
regex: '(endpoint|instance|location|prometheus|prometheus_replica|service|uid)'
- sourceLabels: [name]
action: keep
regex: '(kube_pod_container_resource_requests|kube_pod_container_status_restarts_total|kube_pod_status_phase|kube_pod_info)'

Expected Result

we need only this metrics and labels

Actual Result

instead , its not able to scrape them...

please let us know if the metrics setup is not done properly

Collector version

ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:0.120.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

Log output

Additional context

No response

[bacherfl]

(Triage) Hi @laxmansoma22 - can you please include a complete collector config to make it easier for the maintainers to reproduce this issue? If possible, please also check if you see something in the logs of the collector that might help with troubleshooting this, and include any relevant log outputs in the issue description

[github-actions]

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole @ArthurSens @krajorama. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[laxmansoma22]

(Triage) Hi @laxmansoma22 - can you please include a complete collector config to make it easier for the maintainers to reproduce this issue? If possible, please also check if you see something in the logs of the collector that might help with troubleshooting this, and include any relevant log outputs in the issue description

Hi Bacher - We are using the target allocator to scrape the servicemonitor and trying to keep certain metrics ,but regex template is not working for us....

[laxmansoma22]

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otelc-ta
namespace: opentelemetry-operator
spec:
env:

• name: GOOGLE_APPLICATION_LOGIN
  valueFrom:
  secretKeyRef:
  name: rke-otel-sa
  key: otel-sa.json
• name: GOOGLE_APPLICATION_CREDENTIALS
  value: /otel/otel-sa.json
• name: OTEL_RESOURCE_ATTRIBUTES
  value: "cluster={{ .Values.clusterName }},namespace=opentelemetry-operator,location={{ .Values.region }},project_id= {{ .Values.projectName }}"
  volumeMounts:
  • name: rke-otel-sa
    mountPath: /otel
    readOnly: true
    volumes:
  • name: rke-otel-sa
    secret:
    secretName: rke-otel-sa
    defaultMode: 0777
    mode: statefulset
    serviceAccount: opentelemetry-operator
    targetAllocator:
    enabled: true
    serviceAccount: opentelemetry-operator
    prometheusCR:
    enabled: true
    podMonitorSelector: {}
    serviceMonitorSelector: {}
    resources:
    requests:
    memory: "200Mi"
    cpu: "100m"

config:
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'kubernetes-nodes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- target_label: address
replacement: kubernetes.default.svc:443
- source_labels: [_meta_kubernetes_node_name]
regex: (.+)
target_label: metrics_path
replacement: /api/v1/nodes/$$1/proxy/metrics/cadvisor
# metric_relabel_configs:
# # - action: labeldrop
# # regex: '(image|id|endpoint|instance|job|location|prometheus|prometheus_replica|service|container|name)'
# - sourceLabels: ["name"]
# action: keep
# regex: "container(memory_working_set_bytes|container_cpu_usage_total)"
# replacement: "$$1"
# separator: ";"
kubeletstats:
collection_interval: 10s
auth_type: 'serviceAccount'
endpoint: '${env:K8S_NODE_NAME}:10250'
insecure_skip_verify: true
metric_groups:
- node
- pod
- container
extra_metadata_labels:
- container.id
processors:
filter/drop_metrics_by_name:
metrics:
exclude:
match_type: strict
metric_names:
- container.memory.rss
# filter/keep_metrics_by_name:
# metrics:
# exclude:
# match_type: strict
# metric_names:
# - istio_requests_total
# - istio_request_duration_milliseconds_bucket

  resourcedetection:
    detectors: [env]
  batch:
    send_batch_size: 1000
    timeout: 5s
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8sattributes:
    extract:
      metadata:
      - k8s.node.name
      - k8s.namespace.name
      - k8s.pod.name
    passthrough: false
    pod_association:
    - sources:
      - from: resource_attribute
        name: k8s.pod.ip
    - sources:
      - from: resource_attribute
        name: k8s.pod.uid
    - sources:
      - from: connection
exporters:
  debug:
    verbosity: detailed
  googlemanagedprometheus:
    metric:
      compression: gzip
      add_metric_suffixes: false
      extra_metrics_config:
        enable_target_info: false
    project: {{ .Values.projectName }}
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [memory_limiter, batch, resourcedetection, filter/drop_metrics_by_name,k8sattributes]
      exporters: [googlemanagedprometheus]

[laxmansoma22]

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: envoy-stats-monitor
namespace: opentelemetry-operator
labels:
monitoring: prometheus
release: opentelemetry-operator
spec:
selector:
matchExpressions:
- {key: istio-prometheus-ignore, operator: DoesNotExist}
namespaceSelector:
any: true
jobLabel: envoy-stats
podMetricsEndpoints:

• path: /stats/prometheus
  interval: 30s
  relabelings:
  • action: keep
    sourceLabels: [app]
  • action: keep
    sourceLabels: [__meta_kubernetes_pod_container_name]
    regex: "istio-proxy"
  • action: keep
    sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
  • sourceLabels: [address, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    targetLabel: address
  • action: labeldrop
    regex: "_meta_kubernetes_pod_label(.+)"
  • sourceLabels: [__meta_kubernetes_namespace]
    action: replace
    targetLabel: namespace
  • sourceLabels: [__meta_kubernetes_pod_name]
    action: replace
    targetLabel: pod_name
    metricRelabelings:
  • sourceLabels: ["name"]
    action: keep
    regex: '(istio_requests_total|istio_request_duration_milliseconds_bucket)'

[ArthurSens]

I believe you'll receive better support by reaching out to the target allocator maintainers :)

The repository is https://github.com/open-telemetry/opentelemetry-operator