creating overlay failed /run/ibpod/infra-container

[fansari]

Issue Description

Describe your issue

Steps to reproduce the issue

Steps to reproduce the issue

1. Start PC

Describe the results you received

No root containers are showing up.

So far only workaround is to remove the pod and recreate it.

This creates /run/ibpod/infra-container.

Describe the results you expected

Root containers should start.

The container is configured in /etc/containers/systemd/bind.container.

[Unit]
Description=bind
After=local-fs.target

[Container]
Image=bind
Volume=bind_db:/var/bind
Volume=bind_conf:/etc/bind
Volume=bind_pki:/etc/pki
PodmanArgs=--pod=dns

[Install]
WantedBy=multi-user.target default.target

podman info output

host:
  arch: amd64
  buildahVersion: 1.40.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 93.42
    systemPercent: 1.62
    userPercent: 4.96
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: silverblue
    version: "42"
  eventLogger: journald
  freeLocks: 2025
  hostname: bat.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 60257
      size: 1
    - container_id: 1
      host_id: 260257
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 60257
      size: 1
    - container_id: 1
      host_id: 260257
      size: 65536
  kernel: 6.14.6-300.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2875727872
  memTotal: 8191688704
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.15.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.15.0
    package: netavark-1.15.0-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.15.0
  ociRuntime:
    name: crun
    package: crun-1.21-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/user/60257/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/sbin/pasta
    package: passt-0^20250512.g8ec1341-1.fc42.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/60257/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/sbin/slirp4netns
    package: slirp4netns-1.3.1-2.fc42.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 16781402112
  swapTotal: 16781402112
  uptime: 0h 35m 49.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/fansari/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 2
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/mnt/data/podman/fansari/containers/storage
  graphRootAllocated: 643930521600
  graphRootUsed: 204811022336
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 39
  runRoot: /run/user/60257
  transientStore: false
  volumePath: /var/mnt/data/podman/fansari/containers/storage/volumes
version:
  APIVersion: 5.5.0
  BuildOrigin: Fedora Project
  Built: 1747180800
  BuiltTime: Wed May 14 02:00:00 2025
  GitCommit: 0dbcb51477ee7ab8d3b47d30facf71fc38bb0c98
  GoVersion: go1.24.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.0

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

OS: Fedora Silverblue 42
SELinux is in permissive mode

Additional information

I get these error messages.

May 23 16:58:30 bat.localdomain podman[4960]: 2025-05-23 16:58:30.400679846 +0200 CEST m=+0.054238812 container create f83ab48f3d794238e19f2eafffac51118f587f84210657b699f0ae6756ff2517 (image=localhost/bind:latest, name=systemd-bind, pod_id=bf9ad69e9a2dfc0dc20b77b305a9f6febbaaccb8cdb12dfd2fba929ccfe6d9b1, PODMAN_SYSTEMD_UNIT=bind.service, io.buildah.version=1.32.0)
May 23 16:58:30 bat.localdomain podman[4960]: 2025-05-23 16:58:30.401579423 +0200 CEST m=+0.055138404 container restart ccdf4d5f3d3e18e0494df904ef659a5b8e407b59d93e1be11b829f4a2d665118 (image=, name=bf9ad69e9a2d-infra, pod_id=bf9ad69e9a2dfc0dc20b77b305a9f6febbaaccb8cdb12dfd2fba929ccfe6d9b1)
May 23 16:58:30 bat.localdomain podman[4960]: 2025-05-23 16:58:30.369037915 +0200 CEST m=+0.022596887 image pull 4a4b079e8dc980ffa5b14563e08abaf528e715885f6d39278a57e44cc3a77960 bind
May 23 16:58:30 bat.localdomain bind[4960]: time="2025-05-23T16:58:30+02:00" level=error msg="Starting some container dependencies"
May 23 16:58:30 bat.localdomain bind[4960]: time="2025-05-23T16:58:30+02:00" level=error msg=""rootfs-overlay: creating overlay failed \"/run/libpod/infra-container\": stat /run/libpod/infra-container: no such file or directory""
May 23 16:58:30 bat.localdomain podman[4960]: 2025-05-23 16:58:30.508798454 +0200 CEST m=+0.162357445 container remove f83ab48f3d794238e19f2eafffac51118f587f84210657b699f0ae6756ff2517 (image=localhost/bind:latest, name=systemd-bind, pod_id=bf9ad69e9a2dfc0dc20b77b305a9f6febbaaccb8cdb12dfd2fba929ccfe6d9b1, PODMAN_SYSTEMD_UNIT=bind.service, io.buildah.version=1.32.0)
May 23 16:58:30 bat.localdomain bind[4960]: Error: starting some containers: internal libpod error
May 23 16:58:30 bat.localdomain systemd[1]: bind.service: Main process exited, code=exited, status=126/n/a

[fansari]

Another (better) workaround (which was not necessary before):

[Unit]
Description=bind
After=local-fs.target

[Service]
ExecStartPre=mkdir -p /run/libpod/infra-container

[Container]
Image=bind
Volume=bind_db:/var/bind
Volume=bind_conf:/etc/bind
Volume=bind_pki:/etc/pki
PodmanArgs=--pod=dns

[Install]
WantedBy=multi-user.target default.target

[Luap99]

PodmanArgs=--pod=dns

Note is generally recommend to use pod units to manage pods with quadlet.

---

But of course regardless the bug here is that the new pause image less pods are broken after a reboot as the code to recreate the tmp directory after reboot seems broken.
cc @jankaluza

[fansari]

PodmanArgs=--pod=dns

Note is generally recommend to use pod units to manage pods with quadlet.

But of course regardless the bug here is that the new pause image less pods are broken after a reboot as the code to recreate the tmp directory after reboot seems broken. cc @jankaluza

Can you give an example how I should do it? This is the only way I know so far with pods.

[Luap99]

An example for a pod quadlet? https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html#pod-units-pod
Just need to create a pod unit (i.e. mytest.pod) and then in the container unit have Pod=mytest.pod

[fansari]

Yeah - this works.

Important was to put "After=dns.pod" int the Unit section of the container. Otherwise the container is not starting after boot because the pod is still not there.

[Unit]
Description=dns
After=local-fs.target

[Pod]
PodName=dns
DNS=::1
DNSSearch=localdoamin
Network=podman1
IP=10.90.0.2
IP6=fdc2:4ba9:85d4:f3c3::2

[Install]
WantedBy=multi-user.target default.target

[Unit]
Description=bind
After=dns.pod

[Container]
Image=bind
Volume=bind_db:/var/bind
Volume=bind_conf:/etc/bind
Volume=bind_pki:/etc/pki
PodmanArgs=--pod=dns

[Install]
WantedBy=multi-user.target default.target

[Luap99]

That is not needed, quadlet manages the dependencies but you must use Pod=dns.pod in the container unit or however you called the file. PodmanArgs bypasses quadlets logic as it just appends the args directly.

[fansari]

Yes - I can confirm it works this way.

Unit]
Description=bind

[Container]
Pod=dns.pod
Image=localhost/bind
Volume=bind_db:/var/bind
Volume=bind_conf:/etc/bind
Volume=bind_pki:/etc/pki

[Install]
WantedBy=multi-user.target default.target