Spans don't have K8s attributes info

[TanYuzhen]

Component(s)

No response

What happened?

Description

I use the k8sattributes to inject the pod's name into the span. However, it seems not work.

Steps to Reproduce

I use the Google-microservice-demo as the microservice system(https://github.com/GoogleCloudPlatform/microservices-demo)
And I have the otelcollector.yaml as below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otelcollector
  namespace: online-trace
  labels:
    app: otelcollector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otelcollector
  template:
    metadata:
      labels:
        app: otelcollector
      # 如果在 Istio 环境下，建议禁用 sidecar 注入，确保 Collector 能识别真实 Pod IP
      # annotations:
      #   sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: otelcollector
      terminationGracePeriodSeconds: 5
      volumes:
      - name: config
        configMap:
          name: otel-collector-config
      containers:
      - name: otelcollector
        image: otel/opentelemetry-collector-contrib:0.82.0
        args:
        - "--config=/conf/otel-collector-config.yaml"
        volumeMounts:
        - name: config
          mountPath: /conf
          readOnly: true
        ports:
        - containerPort: 4317   # OTLP gRPC
        - containerPort: 4318   # OTLP HTTP
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        resources:
          requests:
            cpu: 200m
            memory: 180Mi
          limits:
            cpu: 300m
            memory: 300Mi
---
apiVersion: v1
kind: Service
metadata:
  name: otelcollector
  namespace: online-trace
spec:
  type: ClusterIP
  selector:
    app: otelcollector
  ports:
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
  - name: otlp-http
    port: 4318
    targetPort: 4318

And the otel-collector-config.yaml is below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: online-trace
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:   # 接收 gRPC OTLP（默认 4317）
          http:   # 接收 HTTP OTLP（默认 4318）

    processors:
      k8sattributes:
        auth_type: serviceAccount
        passthrough: false
        extract:
          metadata:
            - k8s.namespace.name
            - k8s.deployment.name
            - k8s.statefulset.name
            - k8s.daemonset.name
            - k8s.cronjob.name
            - k8s.job.name
            - k8s.node.name
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.pod.start_time
          labels:
            - tag_name: app.label.version
              key: version
              from: pod
        pod_association:
          - sources:
              - from: resource_attribute
                name: k8s.pod.ip
          - sources:
              - from: resource_attribute
                name: k8s.pod.uid
          - sources:
              - from: resource_attribute
                name: k8s.namespace.name
          - sources:
              - from: resource_attribute
                name: k8s.pod.name
          - sources:
              - from: connection

      batch: {}   # 建议保留 batch 提升吞吐

    exporters:
      jaeger:
        endpoint: "jaeger.online-trace.svc.cluster.local:14250"
        tls:
          insecure: true
      logging:
        verbosity: detailed

    service:
      pipelines:
        traces:
          receivers:  [otlp]
          processors: [k8sattributes, batch]
          exporters:  [jaeger, logging]

And according to the official doc , there needs a RBAC serviceAccount, and the otel-account.yaml is below

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otelcollector
  namespace: online-trace
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otelcollector
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces", "nodes"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["apps"]
  resources: ["replicasets"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources: ["replicasets"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otelcollector
subjects:
- kind: ServiceAccount
  name: otelcollector
  namespace: online-trace
roleRef:
  kind: ClusterRole
  name: otelcollector
  apiGroup: rbac.authorization.k8s.io

And the exporter Jaeger configuration file is below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: online-trace
  labels:
    app: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      serviceAccountName: default
      containers:
      - name: jaeger
        image: jaegertracing/all-in-one:1.35.1
        env:
        - name: COLLECTOR_ZIPKIN_HOST_PORT
          value: ":9411"
        - name: COLLECTOR_OTLP_ENABLED
          value: "true"
        ports:
        - containerPort: 5775
          protocol: UDP
        - containerPort: 6831
          protocol: UDP
        - containerPort: 6832
          protocol: UDP
        - containerPort: 5778
        - containerPort: 14250
        - containerPort: 14268
        - containerPort: 14269
        - containerPort: 16686
        - containerPort: 9411
        - containerPort: 4317
        - containerPort: 4318
        resources:
          requests:
            cpu: 200m
            memory: 180Mi
          limits:
            cpu: 300m
            memory: 300Mi
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger
  namespace: online-trace
spec:
  type: ClusterIP
  selector:
    app: jaeger
  ports:
  - name: "5775"
    port: 5775
    targetPort: 5775
    protocol: UDP
  - name: "6831"
    port: 6831
    targetPort: 6831
    protocol: UDP
  - name: "6832"
    port: 6832
    targetPort: 6832
    protocol: UDP
  - name: "5778"
    port: 5778
    targetPort: 5778
  - name: "14250"
    port: 14250
    targetPort: 14250
  - name: "14268"
    port: 14268
    targetPort: 14268
  - name: "14269"
    port: 14269
    targetPort: 14269
  - name: "9411"
    port: 9411
    targetPort: 9411
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
  - name: otlp-http
    port: 4318
    targetPort: 4318
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger-frontend
  namespace: online-trace
spec:
  type: NodePort
  selector:
    app: jaeger
  ports:
  - name: "16686"
    port: 16686
    targetPort: 16686

Expected Result

I want the span in the jaeger have the pod's attributes such as pod'name and the label of key version

Actual Result

It didn't inject any information I want as below:

Collector version

The collector image I use is otel/opentelemetry-collector-contrib:0.82.0

Environment information

Environment

OS: (e.g., "Ubuntu 22.04")
K8s version: v1.29.14

OpenTelemetry Collector configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: online-trace
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:   # 接收 gRPC OTLP（默认 4317）
          http:   # 接收 HTTP OTLP（默认 4318）

    processors:
      k8sattributes:
        auth_type: serviceAccount
        passthrough: false
        extract:
          metadata:
            - k8s.namespace.name
            - k8s.deployment.name
            - k8s.statefulset.name
            - k8s.daemonset.name
            - k8s.cronjob.name
            - k8s.job.name
            - k8s.node.name
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.pod.start_time
          labels:
            - tag_name: app.label.version
              key: version
              from: pod
        pod_association:
          - sources:
              - from: resource_attribute
                name: k8s.pod.ip
          - sources:
              - from: resource_attribute
                name: k8s.pod.uid
          - sources:
              - from: resource_attribute
                name: k8s.namespace.name
          - sources:
              - from: resource_attribute
                name: k8s.pod.name
          - sources:
              - from: connection

      batch: {}   # 建议保留 batch 提升吞吐

    exporters:
      jaeger:
        endpoint: "jaeger.online-trace.svc.cluster.local:14250"
        tls:
          insecure: true
      logging:
        verbosity: detailed

    service:
      pipelines:
        traces:
          receivers:  [otlp]
          processors: [k8sattributes, batch]
          exporters:  [jaeger, logging]

Log output

Events:
SpanEvent #0
     -> Name: message
     -> Timestamp: 2025-05-16 13:57:34.698757005 +0000 UTC
     -> DroppedAttributesCount: 0
     -> Attributes::
          -> message.id: Int(1)
          -> message.type: Str(RECEIVED)
SpanEvent #1
     -> Name: message
     -> Timestamp: 2025-05-16 13:57:34.888067685 +0000 UTC
     -> DroppedAttributesCount: 0
     -> Attributes::
          -> message.id: Int(2)
          -> message.type: Str(SENT)
	{"kind": "exporter", "data_type": "traces", "name": "logging"}
2025-05-16T13:57:40.419Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 1}
2025-05-16T13:57:40.419Z	info	ResourceSpans #0
Resource SchemaURL: 
Resource attributes:
     -> service.name: Str(cartservice)
     -> service.version: Str(1.0.0)
     -> k8s.pod.ip: Str(127.0.0.6)
ScopeSpans #0
ScopeSpans SchemaURL: 
InstrumentationScope OpenTelemetry.Instrumentation.AspNetCore 1.0.0.0
Span #0
    Trace ID       : b0b1a648440b29a4a18c722597e68523
    Parent ID      : f7dcbc06fc762d96
    ID             : 229d98219a4f9332
    Name           : hipstershop.CartService/GetCart
    Kind           : Server
    Start time     : 2025-05-16 13:57:32.5275849 +0000 UTC
    End time       : 2025-05-16 13:57:33.4296016 +0000 UTC
    Status code    : Unset
    Status message : 
Attributes:
     -> http.host: Str(cartservice:7070)
     -> http.method: Str(POST)
     -> http.target: Str(/hipstershop.CartService/GetCart)
     -> http.url: Str(http://cartservice:7070/hipstershop.CartService/GetCart)
     -> http.user_agent: Str(grpc-go/1.44.1-dev)
     -> PodName: Str(cartservice-v2-5489b98457-7ttdd)
     -> NodeName: Str(serverless-node3)
     -> http.status_code: Int(200)
     -> rpc.system: Str(grpc)
     -> net.peer.ip: Str(::ffff:127.0.0.6)
     -> net.peer.port: Int(60003)
     -> rpc.service: Str(hipstershop.CartService)
     -> rpc.method: Str(GetCart)
     -> rpc.grpc.status_code: Int(0)
	{"kind": "exporter", "data_type": "traces", "name": "logging"}
2025-05-16T13:57:45.833Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 1}
2025-05-16T13:57:45.834Z	info	ResourceSpans #0
Resource SchemaURL: 
Resource attributes:
     -> service.name: Str(cartservice)
     -> service.version: Str(1.0.0)
     -> k8s.pod.ip: Str(127.0.0.6)
ScopeSpans #0
ScopeSpans SchemaURL: 
InstrumentationScope OpenTelemetry.StackExchange.Redis 1.0.0.0
Span #0
    Trace ID       : b0b1a648440b29a4a18c722597e68523
    Parent ID      : 229d98219a4f9332
    ID             : dee7827a6d7feb9e
    Name           : HGET
    Kind           : Client
    Start time     : 2025-05-16 13:57:33.3317792 +0000 UTC
    End time       : 2025-05-16 13:57:33.3387267 +0000 UTC
    Status code    : Unset
    Status message : 
Attributes:
     -> db.system: Str(redis)
     -> db.redis.flags: Str(None)
     -> db.statement: Str(HGET 55621d05-0315-4ee7-bad9-d46045965dcc)
     -> net.peer.name: Str(redis-cart)
     -> net.peer.port: Int(6379)
     -> db.redis.database_index: Int(0)
     -> peer.service: Str(redis-cart:6379)
Events:
SpanEvent #0
     -> Name: Enqueued
     -> Timestamp: 2025-05-16 13:57:33.3365933 +0000 UTC
     -> DroppedAttributesCount: 0
SpanEvent #1
     -> Name: Sent
     -> Timestamp: 2025-05-16 13:57:33.3366423 +0000 UTC
     -> DroppedAttributesCount: 0
SpanEvent #2
     -> Name: ResponseReceived
     -> Timestamp: 2025-05-16 13:57:33.3385748 +0000 UTC
     -> DroppedAttributesCount: 0
	{"kind": "exporter", "data_type": "traces", "name": "logging"}

Additional context

No response

[iblancasa]

Can you check if there is any error in the logs? Maybe some permission is not properly set.

[TanYuzhen]

I follow the offical doc https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor/ to configure the permission of the RBAC and the logs of the otelcollector is too long and I copy it into a txt file, and search error info and search nothing about it.

Can you give me some other possible reason cause the issues?

[github-actions]

Pinging code owners for processor/k8sattributes: @dmitryax @fatsheep9146 @TylerHelmuth @ChrsMark. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[ChrsMark]

@TanYuzhen looking into the pod association rules, the incoming data should match at least one of the required resource attributes (or the incoming connection):

pod_association:
  - sources:
      - from: resource_attribute
        name: k8s.pod.ip
  - sources:
      - from: resource_attribute
        name: k8s.pod.uid
  - sources:
      - from: resource_attribute
        name: k8s.namespace.name
  - sources:
      - from: resource_attribute
        name: k8s.pod.name
  - sources:
      - from: connection

From the shared Span that is not enriched I see that only the k8s.pod.ip resource attribute is present k8s.pod.ip: Str(127.0.0.6) but I wonder if this is actually a valid value. Could you check with kubectl if the target Pod has the same IP in podIP?

[TanYuzhen]

@ChrsMark In fact, the pod's IP isn't 127.0.0.6, I don't know why its k8s.pod.ip was assigned 127.0.0.6

[ChrsMark]

Then that's probably the reason why pod association does not work as expected. The incoming Spans should at least have an existing valid resource_attribute present. Do you know how is this is added to the Spans?

I would also suggest considering enriching your Spans with more k8s related data at first place. Probably using the OTEL_RESOURCE_ATTRIBUTES env var to set k8s related attributes for instrumented Pods. See an example at https://github.com/open-telemetry/opentelemetry-helm-charts/blob/78a064a0ad41cbd31ab37e1ecfede89a9fab59fa/charts/opentelemetry-demo/examples/custom-environment-variables/rendered/component.yaml#L489.

[TanYuzhen]

OK, thanks. But I still want to know why the IP of the pod was assigned to 127.0.0.6?

[ChrsMark]

That's a question I also had :). Since no enrichment takes place the Spans come into the Collector having already this attributes. So it should be set from a component outside of the Collector? Maybe the SDK?