Rootless containers sequencing/race condition

[bexelbie]

Description

When using rootless containers created via podman quadlets the container execution is triggered by systemd --user before the host networking is up. This is likely a sequence of system targets issue or a race condition.

Impact

Running rootless containers fails because the containers cannot be pulled.

Environment and steps to reproduce

1. Set-up:

A system has been setup with a user name sleeper who has linger enabled and is not a sudoer. They have a podman quadlet that touches a file name datetime in their home directory.

2. Task: Run a rootless container

3. Action(s): Bug is triggered because the rootless container cannot be pulled from the registry

4. Error: container doesn't execute

Expected behavior

/home/sleeper/datetime should exist

Additional information

Please add any information here that does not fit the above format.

Here is an ignition file that generates the failure:

variant: flatcar
version: 1.1.0

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ...
    - name: sleeper # rootless containers
      ssh_authorized_keys:
        - ...
      shell: /bin/bash

storage:
  files:
    # Add Podman and configure it
    - path: /etc/containers/policy.json
      contents:
        source: https://raw.githubusercontent.com/containers/podman/main/test/policy.json
    - path: /etc/flatcar/enabled-sysext.conf
      contents:
        inline: |
          podman
    - path: /etc/containers/registries.conf
      contents:
        inline: |
         # Add default registries to search
         unqualified-search-registries = ['docker.io', 'quay.io', 'registry.fedoraproject.org']
    - path: /etc/subuid # Manually update /etc/?id ... note that ignition is ignoring newlines unless escaped with a \ ... not a \n
      overwrite: true
      contents:
        inline: |
          core:100000:65536\
          sleeper:300000:65536
      mode: 0644
    - path: /etc/subgid
      overwrite: true
      contents:
        inline: |
          core:100000:65536\
          sleeper:300000:65536
      mode: 0644

    # Set up linger for our rootless containers user
    - path: /var/lib/systemd/linger/sleeper
      mode: 0644

    # container test
    - path: /home/sleeper/.config/containers/systemd/sleeper-test.container
      mode: 0644
      contents:
        inline: |
          [Unit]
          Description=Create a timestamped file in the user's home directory
          
          [Container]
          Image=docker.io/library/alpine:latest
          Volume=%h:/mnt/hosthome
          Exec=/bin/touch /mnt/hosthome/datefile
          
          [Service]
          Restart=no
          
          [Install]
          WantedBy=default.target
      user:
        name: sleeper
      group:
        name: sleeper

  links:
    # Remove docker and containerd
    - path: /etc/extensions/docker-flatcar.raw
      target: /dev/null
      overwrite: true
    - path: /etc/extensions/containerd-flatcar.raw
      target: /dev/null
      overwrite: true

  directories:
    # Create the systemd-user directory for rootless containers - sadly this can't be done in one step
    # Adapted from: https://docs.fedoraproject.org/en-US/fedora-coreos/tutorial-user-systemd-unit-on-boot/
    - path: /home/sleeper/.config
      mode: 0755
      user:
        name: sleeper
      group:
        name: sleeper
    - path: /home/sleeper/.config/systemd
      mode: 0755
      user:
        name: sleeper
      group:
        name: sleeper
    - path: /home/sleeper/.config/systemd/user
      mode: 0755
      user:
        name: sleeper
      group:
        name: sleeper
    - path: /home/sleeper/.config/systemd/user/default.target.wants
      mode: 0755
      user:
        name: sleeper
      group:
        name: sleeper
    - path: /home/sleeper/.config/containers/systemd
      mode: 0755
      user:
        name: sleeper
      group:
        name: sleeper

This ignition file works, by creating a workaround. A system level systemd service waits for the network-online target and touches /run/user/0/network-ready. A systemd --user service continuously tries for the path /run/user/0/network-ready. This unit is the condition for the podman quadlet.

variant: flatcar
version: 1.1.0

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC1N+xhi9y/rHURF3P0c6TiEGizwFnTBKH5GbQI46uyb
    - name: sleeper # rootless containers
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC1N+xhi9y/rHURF3P0c6TiEGizwFnTBKH5GbQI46uyb
      shell: /bin/bash

storage:
  files:
    # Add Podman and configure it
    - path: /etc/containers/policy.json
      contents:
        source: https://raw.githubusercontent.com/containers/podman/main/test/policy.json
    - path: /etc/flatcar/enabled-sysext.conf
      contents:
        inline: |
          podman
    - path: /etc/containers/registries.conf
      contents:
        inline: |
         # Add default registries to search
         unqualified-search-registries = ['docker.io', 'quay.io', 'registry.fedoraproject.org']
    - path: /etc/subuid # Manually update /etc/?id ... note that ignition is ignoring newlines unless escaped with a \ ... not a \n
      overwrite: true
      contents:
        inline: |
          core:100000:65536\
          sleeper:300000:65536
      mode: 0644
    - path: /etc/subgid
      overwrite: true
      contents:
        inline: |
          core:100000:65536\
          sleeper:300000:65536
      mode: 0644

    # Set up linger for our rootless containers user
    - path: /var/lib/systemd/linger/sleeper
      mode: 0644

    # container test
    - path: /home/sleeper/.config/containers/systemd/sleeper-test.container
      mode: 0644
      contents:
        inline: |
          [Unit]
          Description=Create a timestamped file in the user's home directory
          After=network-ready.service
          Requires=network-ready.service
          
          [Container]
          Image=docker.io/library/alpine:latest
          Volume=%h:/mnt/hosthome
          Exec=/bin/touch /mnt/hosthome/datefile
          
          [Service]
          Restart=no
          
          [Install]
          WantedBy=default.target
      user:
        name: sleeper
      group:
        name: sleeper

    # Network Ready indicator user level for sleeper and rootless containers
    #   This gets the race condition where systemd --user is started before the network is ready
    #   --user cannot rely on system level network targets
    - path: /home/sleeper/.config/systemd/user/network-ready.service
      mode: 0644
      contents:
        inline: |
          [Unit]
          Description=Wait for network ready
          
          [Service]
          Type=oneshot
          ExecStart=/bin/sh -c 'while [ ! -e /run/user/0/network-ready ]; do sleep 5; done'
          RemainAfterExit=yes
          
          [Install]
          WantedBy=default.target
      user:
        name: sleeper
      group:
        name: sleeper

  links:
    # Network Ready Service (sleeper user) - enable unit manually
    #   This gets the race condition where systemd --user is started before the network is ready
    #   --user cannot rely on system level network targets
    - path: /home/sleeper/.config/systemd/user/default.target.wants/network-ready.service
      target: /home/sleeper/.config/systemd/user/network-ready.service
      overwrite: true

    # Remove docker and containerd
    - path: /etc/extensions/docker-flatcar.raw
      target: /dev/null
      overwrite: true
    - path: /etc/extensions/containerd-flatcar.raw
      target: /dev/null
      overwrite: true

  directories:
    # Create the systemd-user directory for rootless containers - sadly this can't be done in one step
    # Adapted from: https://docs.fedoraproject.org/en-US/fedora-coreos/tutorial-user-systemd-unit-on-boot/
    - path: /home/sleeper/.config
      mode: 0755
      user:
        name: sleeper
      group:
        name: sleeper
    - path: /home/sleeper/.config/systemd
      mode: 0755
      user:
        name: sleeper
      group:
        name: sleeper
    - path: /home/sleeper/.config/systemd/user
      mode: 0755
      user:
        name: sleeper
      group:
        name: sleeper
    - path: /home/sleeper/.config/systemd/user/default.target.wants
      mode: 0755
      user:
        name: sleeper
      group:
        name: sleeper
    - path: /home/sleeper/.config/containers/systemd
      mode: 0755
      user:
        name: sleeper
      group:
        name: sleeper

systemd:
  units:
    # Network Ready indicator for systemd --user users
    #   This gets the race condition where systemd --user is started before the network is ready
    #   --user cannot rely on system level network targets
    - name: network-ready.service
      enabled: true
      contents: |
        [Unit]
        Description=Notify user session that network is ready
        After=network-online.target
        
        [Service]
        Type=oneshot
        ExecStart=/bin/sh -c "mkdir -p /run/user/%U; touch /run/user/%U/network-ready"
        
        [Install]
        WantedBy=network-online.target

Here is failure output from journactl

Mar 03 13:35:47 localhost (systemd)[2312]: pam_unix(systemd-user:session): session opened for user sleeper(uid=1000) by (uid=0)
Mar 03 13:35:48 localhost sleeper-test[2333]: Trying to pull docker.io/library/alpine:latest...
Mar 03 13:35:48 localhost sleeper-test[2333]: Pulling image //alpine:latest inside systemd: setting pull timeout to 5m0s
Mar 03 13:35:48 localhost sleeper-test[2333]: Error: initializing source docker://alpine:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution

Here is the sequence of systemd targets reached

Notice:

• user-config is reached at 13:35:47
• The user systemd sessions starts at 13:35:47 and reaches default.target at 13:35:48
• network-online does not occur until 13:35:52
• multi-user does not occur until 13:35:52

Mar 03 13:35:14 localhost systemd[1]: Reached target cryptsetup-pre.target - Local Encrypted Volumes (Pre).
Mar 03 13:35:14 localhost systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes.
Mar 03 13:35:14 localhost systemd[1]: Reached target paths.target - Path Units.
Mar 03 13:35:14 localhost systemd[1]: Reached target slices.target - Slice Units.
Mar 03 13:35:14 localhost systemd[1]: Reached target swap.target - Swaps.
Mar 03 13:35:14 localhost systemd[1]: Reached target timers.target - Timer Units.
Mar 03 13:35:14 localhost systemd[1]: Reached target sockets.target - Socket Units.
Mar 03 13:35:14 localhost systemd[1]: Reached target nss-lookup.target - Host and Network Name Lookups.
Mar 03 13:35:14 localhost systemd[1]: Reached target remote-fs-pre.target - Preparation for Remote File Systems.
Mar 03 13:35:14 localhost systemd[1]: Reached target remote-cryptsetup.target - Remote Encrypted Volumes.
Mar 03 13:35:14 localhost systemd[1]: Reached target remote-fs.target - Remote File Systems.
Mar 03 13:35:15 localhost systemd[1]: Reached target network.target - Network.
Mar 03 13:35:15 localhost systemd[1]: Reached target initrd-root-device.target - Initrd Root Device.
Mar 03 13:35:15 localhost systemd[1]: Reached target local-fs-pre.target - Preparation for Local File Systems.
Mar 03 13:35:15 localhost systemd[1]: Reached target local-fs.target - Local File Systems.
Mar 03 13:35:15 localhost systemd[1]: Reached target sysinit.target - System Initialization.
Mar 03 13:35:15 localhost systemd[1]: Reached target basic.target - Basic System.
Mar 03 13:35:16 localhost systemd[1]: Reached target initrd-root-fs.target - Initrd Root File System.
Mar 03 13:35:16 localhost systemd[1]: Reached target ignition-diskful.target - Ignition Boot Disk Setup.
Mar 03 13:35:44 localhost systemd[1]: Reached target ignition-complete.target - Ignition Complete.
Mar 03 13:35:44 localhost systemd[1]: Reached target initrd-fs.target - Initrd File Systems.
Mar 03 13:35:44 localhost systemd[1]: Reached target initrd.target - Initrd Default Target.
Mar 03 13:35:44 localhost systemd[1]: Reached target initrd-switch-root.target - Switch Root.
Mar 03 13:35:14 localhost systemd[1]: Reached target cryptsetup-pre.target - Local Encrypted Volumes (Pre).
Mar 03 13:35:14 localhost systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes.
Mar 03 13:35:14 localhost systemd[1]: Reached target paths.target - Path Units.
Mar 03 13:35:14 localhost systemd[1]: Reached target slices.target - Slice Units.
Mar 03 13:35:14 localhost systemd[1]: Reached target swap.target - Swaps.
Mar 03 13:35:14 localhost systemd[1]: Reached target timers.target - Timer Units.
Mar 03 13:35:14 localhost systemd[1]: Reached target sockets.target - Socket Units.
Mar 03 13:35:14 localhost systemd[1]: Reached target nss-lookup.target - Host and Network Name Lookups.
Mar 03 13:35:14 localhost systemd[1]: Reached target remote-fs-pre.target - Preparation for Remote File Systems.
Mar 03 13:35:14 localhost systemd[1]: Reached target remote-cryptsetup.target - Remote Encrypted Volumes.
Mar 03 13:35:14 localhost systemd[1]: Reached target remote-fs.target - Remote File Systems.
Mar 03 13:35:15 localhost systemd[1]: Reached target network.target - Network.
Mar 03 13:35:15 localhost systemd[1]: Reached target initrd-root-device.target - Initrd Root Device.
Mar 03 13:35:15 localhost systemd[1]: Reached target local-fs-pre.target - Preparation for Local File Systems.
Mar 03 13:35:15 localhost systemd[1]: Reached target local-fs.target - Local File Systems.
Mar 03 13:35:15 localhost systemd[1]: Reached target sysinit.target - System Initialization.
Mar 03 13:35:15 localhost systemd[1]: Reached target basic.target - Basic System.
Mar 03 13:35:16 localhost systemd[1]: Reached target initrd-root-fs.target - Initrd Root File System.
Mar 03 13:35:16 localhost systemd[1]: Reached target ignition-diskful.target - Ignition Boot Disk Setup.
Mar 03 13:35:44 localhost systemd[1]: Reached target ignition-complete.target - Ignition Complete.
Mar 03 13:35:44 localhost systemd[1]: Reached target initrd-fs.target - Initrd File Systems.
Mar 03 13:35:44 localhost systemd[1]: Reached target initrd.target - Initrd Default Target.
Mar 03 13:35:44 localhost systemd[1]: Reached target initrd-switch-root.target - Switch Root.
Mar 03 13:35:45 localhost systemd[1]: Reached target cryptsetup-pre.target - Local Encrypted Volumes (Pre).
Mar 03 13:35:45 localhost systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.
Mar 03 13:35:45 localhost systemd[1]: Reached target remote-cryptsetup.target - Remote Encrypted Volumes.
Mar 03 13:35:45 localhost systemd[1]: Reached target remote-fs.target - Remote File Systems.
Mar 03 13:35:45 localhost systemd[1]: Reached target slices.target - Slice Units.
Mar 03 13:35:45 localhost systemd[1]: Reached target swap.target - Swaps.
Mar 03 13:35:45 localhost systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.
Mar 03 13:35:45 localhost systemd[1]: Reached target machines.target - Containers.
Mar 03 13:35:45 localhost systemd[1]: Reached target network-pre.target - Preparation for Network.
Mar 03 13:35:45 localhost systemd[1]: Reached target local-fs.target - Local File Systems.
Mar 03 13:35:45 localhost systemd[1]: Reached target first-boot-complete.target - First Boot Complete.
Mar 03 13:35:46 localhost systemd[1]: Reached target nss-lookup.target - Host and Network Name Lookups.
Mar 03 13:35:46 localhost systemd[1]: Reached target time-set.target - System Time Set.
Mar 03 13:35:46 localhost systemd[1]: Reached target network.target - Network.
Mar 03 13:35:47 localhost systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes.
Mar 03 13:35:47 localhost systemd[1]: Reached target sysinit.target - System Initialization.
Mar 03 13:35:47 localhost systemd[1]: Reached target paths.target - Path Units.
Mar 03 13:35:47 localhost systemd[1]: Reached target timers.target - Timer Units.
Mar 03 13:35:47 localhost systemd[1]: Reached target sockets.target - Socket Units.
Mar 03 13:35:47 localhost systemd[1]: Reached target basic.target - Basic System.
Mar 03 13:35:47 localhost systemd[1]: Reached target system-config.target - Load system-provided cloud configs.
Mar 03 13:35:47 localhost systemd[1]: Reached target user-config.target - Load user-provided cloud configs.
Mar 03 13:35:47 localhost systemd[1]: Reached target getty.target - Login Prompts.
Mar 03 13:35:47 localhost (systemd)[2312]: pam_unix(systemd-user:session): session opened for user sleeper(uid=1000) by (uid=0)
Mar 03 13:35:47 localhost systemd[2312]: Queued start job for default target default.target.
Mar 03 13:35:47 localhost systemd[2312]: Created slice app.slice - User Application Slice.
Mar 03 13:35:47 localhost systemd[2312]: Reached target paths.target - Paths.
Mar 03 13:35:47 localhost systemd[2312]: Reached target timers.target - Timers.
Mar 03 13:35:47 localhost systemd[2312]: Starting dbus.socket - D-Bus User Message Bus Socket...
Mar 03 13:35:47 localhost systemd[2312]: Listening on dbus.socket - D-Bus User Message Bus Socket.
Mar 03 13:35:47 localhost systemd[2312]: Reached target sockets.target - Sockets.
Mar 03 13:35:47 localhost systemd[2312]: Reached target basic.target - Basic System.
Mar 03 13:35:47 localhost systemd[2312]: Starting sleeper-test.service - Create a timestamped file in the user's home directory...
Mar 03 13:35:48 localhost systemd[2312]: Created slice session.slice - User Core Session Slice.
Mar 03 13:35:48 localhost systemd[2312]: Starting dbus.service - D-Bus User Message Bus...
Mar 03 13:35:48 localhost systemd[2312]: Started dbus.service - D-Bus User Message Bus.
Mar 03 13:35:48 localhost systemd[2312]: Created slice user.slice - Slice /user.
Mar 03 13:35:48 localhost systemd[2312]: Started podman-pause-544bf159.scope.
Mar 03 13:35:48 localhost systemd[2312]: sleeper-test.service: Main process exited, code=exited, status=125/n/a
Mar 03 13:35:48 localhost systemd[2312]: sleeper-test.service: Failed with result 'exit-code'.
Mar 03 13:35:48 localhost systemd[2312]: Failed to start sleeper-test.service - Create a timestamped file in the user's home directory.
Mar 03 13:35:48 localhost systemd[2312]: Reached target default.target - Main User Target.
Mar 03 13:35:48 localhost systemd[2312]: Startup finished in 549ms.
Mar 03 13:35:52 localhost systemd[1]: Reached target network-online.target - Network is Online.
Mar 03 13:35:52 localhost systemd[1]: Reached target multi-user.target - Multi-User System.
Mar 03 13:35:55 localhost systemd[2378]: Reached target paths.target - Paths.
Mar 03 13:35:55 localhost systemd[2378]: Reached target timers.target - Timers.
Mar 03 13:35:55 localhost systemd[2378]: Reached target sockets.target - Sockets.
Mar 03 13:35:55 localhost systemd[2378]: Reached target basic.target - Basic System.
Mar 03 13:35:55 localhost systemd[2378]: Reached target default.target - Main User Target.
Mar 03 13:36:08 localhost systemd[2312]: Started podman-2434.scope.

[bexelbie]

I did some more research and I am going to suggest this become a point of documentation. I believe, though I encourage others to consider and verify, that this is expected behavior.

To help others who follow, this override will move the systemd-user sessions to after the network start. It affects all users on the system.

systemd:
  units:
    - name: systemd-user-sessions.service
      dropins:
        - name: users-after-network.conf
          contents: |
            [Unit]
            After=network-online.target

I have proposed a documentation PR to save this information in a more durable way.

[t-lo]

Thank you for the docs PR, @bexelbie ! Just merged it. Do you think we can close this issue?

[bexelbie]

absolutely