Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-remount-fs.service fail with composefs enabled #3193

Closed
karuboniru opened this issue Feb 22, 2024 · 11 comments · May be fixed by #3372
Closed

systemd-remount-fs.service fail with composefs enabled #3193

karuboniru opened this issue Feb 22, 2024 · 11 comments · May be fixed by #3372
Labels
area/composefs Issues related to composefs difficulty/medium medium complexity/difficutly issue reward/small This is a minor fix or cleanup triaged This issue has been evaluated and is valid

Comments

@karuboniru
Copy link

  • Enable composefs by adding following lines to ostree-prepare-root.conf:
[composefs]
enabled=yes
  • Apply image and reboot
  • Check systemctl status systemd-remount-fs.service

Result:

2月 22 13:28:48 karubook systemd-remount-fs[19855]: mount: /: mount point not mounted or bad option.
2月 22 13:28:48 karubook systemd-remount-fs[19855]:        dmesg(1) may have more information after failed mount system call.
2月 22 13:28:48 karubook systemd-remount-fs[19854]: /usr/bin/mount for / exited with exit status 32.
2月 22 13:28:48 karubook systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE

This might be a systemd bug that tried to re-mount the overlay on / type overlay (ro,relatime,seclabel,lowerdir=/run/ostree/.private/cfsroot-lower::/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on). If you can comfirm this I can file an another issue on systemd repo.

@karuboniru
Copy link
Author

Cleaning the / entry in /etc/fstab can workaround this, while this introduced another issue where during shutdown there is a [Failed] Failed unmounting sysroot.mount - /sysroot without any information being given in journal.

@cgwalters
Copy link
Member

What's in your /etc/fstab?

@cgwalters cgwalters added needinfo This issue needs more information from the reporter area/composefs Issues related to composefs labels Feb 23, 2024
@karuboniru
Copy link
Author

# UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 /                     btrfs   subvol=root,compress=zstd:1,noatime,user_subvol_rm_allowed,ro 0 0
UUID=f35b55c6-f8c3-4279-be97-8811abda483a /boot                   ext4    defaults        1 2
UUID=6907-17CA          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 /var                    btrfs   subvol=var,compress=zstd:1,noatime,user_subvol_rm_allowed 0 0
UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 /var/home               btrfs   subvol=home,user_subvol_rm_allowed,noatime     0 0

I understand that the remount failed due to / no longer being a "naked btrfs mount` but now an overlayfs. So I commented it out.

@cgwalters
Copy link
Member

The general fix here is to use rw in your kernel commandline. Hmm, actually you have ro in your mount options...is that intentional? What's in your /proc/cmdline? Was your rootfs being mounted rw before? You have writable subvolumes, but still...

@karuboniru
Copy link
Author

actually you have ro in your mount options...is that intentional

Sorry, that should be added during the process when I am trying to workaround the issue myself. I first tried to add the ro and it did not work then I commented the whole line out.

The cmdline is

BOOT_IMAGE=(hd0,gpt2)/ostree/fedora-0ce8c3b357a4e48a8c5b1617d75fbdb00219f87679fc273acb00dd831cddd07d/vmlinuz-6.8.0-0.rc5.41.fc40.x86_64 rhgb quiet root=UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 rootflags=subvol=root rw ostree=/ostree/boot.0/fedora/0ce8c3b357a4e48a8c5b1617d75fbdb00219f87679fc273acb00dd831cddd07d/0

@cgwalters
Copy link
Member

I think we'll need to patch systemd for this most likely; there's a bit of a conflict here started from the basic fact that composefs is inherently an overlayfs on top of the underlying real root, and systemd-remount-fs is unaware of this.

Let's take as a prerequisite that we have an elegant way for external tooling to "peel" the overlayfs and find the backing filesystem - singular. In theory a composefs (as it's an overlayfs) can span multiple filesystems, but I struggle to think of a realistic use case for that - at least for the root filesystem.

The systemd volatile root logic has an ad-hoc symlink it creates...would be nicer to be able to attach this as in-band metadata on the mount somehow. But I guess for our use case, similar to the systemd one we're just handling the single root filesystem so a single well-known symlink is probably OK for now.

We then have a tension around the writability of the underlying real root vs the composefs. This issue relates to #3177 a lot.

I'd still take the basic stance that it just makes sense to mount the real root writable (if that's the intent) from the initramfs from the start.

Probably the most practical change is for systemd-remount-fs to gain some logic for detecting that the root it sees is an overlayfs, and if the backing fileystem matches that defined in /etc/fstab, it can handle remounting that instead. And in the common/expected case, the writability and mount options should match and it should be a no-op.

@cgwalters cgwalters added difficulty/medium medium complexity/difficutly issue triaged This issue has been evaluated and is valid reward/small This is a minor fix or cleanup and removed needinfo This issue needs more information from the reporter labels Feb 26, 2024
cgwalters added a commit to cgwalters/bootc that referenced this issue Mar 22, 2024
This is a giant and hacky workaround for
ostreedev/ostree#3193

The better real fix is probably in either systemd or anaconda
(more realistically both) but let's paper over things here for now.

Having code to run as a generator will likely be useful in the
future anyways.

Signed-off-by: Colin Walters <[email protected]>
cgwalters added a commit to cgwalters/bootc that referenced this issue Mar 22, 2024
This is a giant and hacky workaround for
ostreedev/ostree#3193

The better real fix is probably in either systemd or anaconda
(more realistically both) but let's paper over things here for now.

Having code to run as a generator will likely be useful in the
future anyways.

Signed-off-by: Colin Walters <[email protected]>
@cgwalters
Copy link
Member

Some code for this in bootc-dev/bootc#417 - but as I said there it's really papering over work we need to do probably in both anaconda and systemd.

cgwalters added a commit to cgwalters/bootc that referenced this issue Mar 22, 2024
This is a giant and hacky workaround for
ostreedev/ostree#3193

The better real fix is probably in either systemd or anaconda
(more realistically both) but let's paper over things here for now.

Having code to run as a generator will likely be useful in the
future anyways.

Signed-off-by: Colin Walters <[email protected]>
cgwalters added a commit to cgwalters/bootc that referenced this issue Mar 22, 2024
This is a giant and hacky workaround for
ostreedev/ostree#3193

The better real fix is probably in either systemd or anaconda
(more realistically both) but let's paper over things here for now.

Having code to run as a generator will likely be useful in the
future anyways.

Signed-off-by: Colin Walters <[email protected]>
@keszybz
Copy link

keszybz commented Jan 8, 2025

I'd still take the basic stance that it just makes sense to mount the real root writable (if that's the intent) from the initramfs from the start.

Yeah, the ro that we still use by default stopped making sense at least a decade ago. It just causes problems and slows down boot.

@cgwalters
Copy link
Member

I forgot to xref here but bootc-dev/bootc#972 landed ➡️
https://containers.github.io/bootc/bootc-install.html#finding-and-configuring-the-physical-root-filesystem

which TL;DR says to use root= and rootflags= kargs and don't have / in /etc/fstab at all, and that's what I think is the best fix for this.

bootc-dev/bootc#971 (comment)
has links to two other trackers.

So I am closing this here, as I now consider this not an ostree (or composefs) problem directly.

That said @keszybz since you are here I am curious if you have any systemd perspective on the previous comment - it'd clearly be nice if systemd-remount-fs did some graceful handling of this, but again as far as I can see it's really best to just use kargs for this as that's the best way to reliably ensure the rootfs is mounted the right way from the start.

@cgwalters cgwalters closed this as not planned Won't fix, can't repro, duplicate, stale Jan 8, 2025
@keszybz
Copy link

keszybz commented Jan 11, 2025

Probably the most practical change is for systemd-remount-fs to gain some logic for detecting that the root it sees is an overlayfs, and if the backing fileystem matches that defined in /etc/fstab, it can handle remounting that instead. And in the common/expected case, the writability and mount options should match and it should be a no-op.

Maybe. Or maybe we should just not touch the root file system if the mount "device" or file system type doesn't match what is listed in /etc/fstab. Looking briefly at the code, we just mount -o remount / unconditionally.

keszybz added a commit to keszybz/systemd that referenced this issue Mar 26, 2025
…ferent is mounted

As reported in ostreedev/ostree#3193, when the system
has mounted a different file system on "/" than what is configured in
/etc/fstab, we still try to execute "mount -o remount /", and that fails because
the filesystem that is actually mounted doesn't know about the options relevant
for the filesystem listed in /etc/fstab. I don't think it's useful to try to
apply options from /etc/fstab if the file system type or device are different.
In fact, it could be dangerous or destructive. If the user mounted something
different than what the fstab specifies, no matter how exactly, we should leave
that mount point alone.
@keszybz
Copy link

keszybz commented Mar 26, 2025

systemd/systemd#36867

keszybz added a commit to keszybz/systemd that referenced this issue Mar 26, 2025
…ferent is mounted

As reported in ostreedev/ostree#3193, when the system
has mounted a different file system on "/" than what is configured in
/etc/fstab, we still try to execute "mount -o remount /", and that fails because
the filesystem that is actually mounted doesn't know about the options relevant
for the filesystem listed in /etc/fstab. I don't think it's useful to try to
apply options from /etc/fstab if the file system type or device are different.
In fact, it could be dangerous or destructive. If the user mounted something
different than what the fstab specifies, no matter by what means, we should leave
that mount point alone.

If the filesystem is (potentially) multi-device, we only require the file
system type to match.
keszybz added a commit to keszybz/systemd that referenced this issue Mar 26, 2025
…ferent is mounted

As reported in ostreedev/ostree#3193, when the system
has mounted a different file system on "/" than what is configured in
/etc/fstab, we still try to execute "mount -o remount /", and that fails because
the filesystem that is actually mounted doesn't know about the options relevant
for the filesystem listed in /etc/fstab. I don't think it's useful to try to
apply options from /etc/fstab if the file system type or device are different.
In fact, it could be dangerous or destructive. If the user mounted something
different than what the fstab specifies, no matter by what means, we should leave
that mount point alone.

If the filesystem is (potentially) multi-device, we only require the file
system type to match.
keszybz added a commit to keszybz/systemd that referenced this issue Apr 1, 2025
As reported in ostreedev/ostree#3193, when the system
has mounted a different file system on "/" than what is configured in
/etc/fstab, we still try to execute "mount -o remount /", and that fails because
the filesystem that is actually mounted doesn't know about the options relevant
for the filesystem listed in /etc/fstab.

Let's be nice to the user and give a hint if the filesystem specified in fstab
doesn't match what is actually mounted. We still want to use the specified
configuration for backwards compatibility. But the mismatch is a potential
problem, for example when the user forgot to update fstab after updating the
boot option or the other way around. By warning, we give the user the chance to
notice this, and for example avoid pasting the (non-working) config from fstab
into a future boot entry.

If the filesystem is (potentially) multi-device, we only require the file
system type to match.

For my system:
$ SYSTEMD_LOG_LEVEL=debug build/systemd-remount-fs
...
Note: file system type mounted at "/" doesn't match fstab (type btrfs vs. vfat).
Consider adjusting fstab.
Remounting /...
...

$ SYSTEMD_LOG_LEVEL=debug build/systemd-remount-fs
...
Mount device /dev/disk/by-uuid/a9194733-9c5d-4fc6-8342-6c50646a4f58 specified in fstab doesn't exist,
             /dev/mapper/luks-4481d273-e5d8-4caa-bea7-d8ac9107dae7 is mounted instead.
Consider adjusting fstab.
Remounting /...
Successfully forked off '(remount)' as PID 482420.
mount: /: can't find UUID=a9194733-9c5d-4fc6-8342-6c50646a4f58.
mount: (hint) your fstab has been modified, but systemd still uses
       the old version; use 'systemctl daemon-reload' to reload.
/usr/bin/mount for / exited with exit status 1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/composefs Issues related to composefs difficulty/medium medium complexity/difficutly issue reward/small This is a minor fix or cleanup triaged This issue has been evaluated and is valid
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants