Failed to create containerd task: Exec: "/otelcontribcol": permission denied: unknown - when running the container as non root

[arundathi-nirmata]

We are running otel-collector deployment on cluster v1.21.1 with nonroot specific security context.

Facing issue of runContainerError and eventually pod goes to CrashLoopBackOff with the below error

Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/otelcontribcol": permission denied: unknown

Below is the deployment I am using

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-agent
  labels:
    app: opentelemetry
    component: otel-agent
spec:
  selector:
    matchLabels:
      app: opentelemetry
      component: otel-agent
  template:
    metadata:
      labels:
        app: opentelemetry
        component: otel-agent
    spec:
      containers:
      - name: otel-agent
        image: otel/opentelemetry-collector-contrib:0.37.1
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsGroup: 10001
          runAsNonRoot: true
          runAsUser: 10001
        livenessProbe:
          exec:
            command:
            - cat
            - /etc/otel/config.yaml
        readinessProbe:
          exec:
            command:
            - cat
            - /etc/otel/config.yaml
        volumeMounts:
        - mountPath: /var/log
          name: varlog
          readOnly: true
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
        - mountPath: /etc/otel/config.yaml
          name: data
          subPath: config.yaml
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: data
        configMap:
          name: otel-agent-config

[rmtobin]

This also is an issue when attempting to run the demo locally (latest version of Docker for Mac) as provided in the documentation.

Running the demo command as written in the docs errors out with ERROR: for otel-collector Cannot start service otel-collector: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/otelcontribcol": permission denied: unknown.

[jcadavez]

Same here, having trouble running it on Github Actions on an Ubuntu runner.

What's the latest version of this Docker image I can use temporarily?

[indrekj]

We were fine with: 0.37.1 and 0.38.0, but when we changed to v0.39.0 we started getting:

Error: failed to start container "otc-container": Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/otelcontribcol": permission denied: unknown

[indrekj]

Inspected the filesystems of both docker images 0.38.0 and v0.39.0.

In 0.38.0:

-rwxr-xr-x  1 indrek indrek 143172784 Nov  1 21:41 otelcontribcol

In v0.39.0:

-rw-r--r-- 1 indrek indrek 143808977 Nov 16 02:54 otelcontribcol

So it looks that otelcontribcol is not marked as executable in v0.39.0 docker image.

Running make docker-otelcontribcol on main branch seemed to build a docker image with executable file correctly, so not sure what happened with the v0.39.0 build.

[NathanielRN]

Should probably be closed as a duplicate of #6343