Discussion: Use overlay mounts instead of copying

[mcassaniti]

There is a lot of copying going on in mkosi and a fair bit of it is fixed data that won't change. Rather than copy files around, why don't we use overlay mounts much more often. This would mean at least the following would become a part of a lower directory rather than a copy:

• Base Trees
• Skeleton Trees
• Any directory caches (see resue_cache())
• Extra Trees

If this a bad idea then let's document the reason(s) why here. The reasons are:

• Makes supporting other caching options such as Support a caching mode where we always append to the existing caches #1749 easier
• Reduces disk I/O and therefore build times
• Should make differencing between images much easier even though we don't do that yet

If there's no major objection I'll clobber together a PR. If we just want to see where it leads to I can get a PR together for that as well.

The challenges are:

• Any of the trees options might be a TAR archive which needs extracting first
• This needs to play nice with overlay, but it's possible to fit in reasonably well
• It might make sense to make this an option rather than enabled for good since OverlayFS comes with some limitations

[DaanDeMeyer]

I'll take a look at any PRs but this problem is already solved by using a filesystem on the host that supports subvolumes or reflinks such as btrfs or xfs.

[mcassaniti]

Reflinks are great if the FS supports them and if everything is on the same file system. I should add to the docs that it's highly recommended to use XFS or btrfs to run mkosi on top of.

[DaanDeMeyer]

Reflinks are great if the FS supports them and if everything is on the same file system. I should add to the docs that it's highly recommended to use XFS or btrfs to run mkosi on top of.

We try to not leave the filesystem that mkosi is invoked from to make reflinks work as much as possible. A documentation update mentioning that COW filesystems are recommended would be much appreciated

[mcassaniti]

It's a bit of an aside, but how might systemd-repart be run to also make sure it uses the same file system?

[DaanDeMeyer]

Not possible unfortunately without mounting the filesystem which means root privileges. If we'd be OK with root I'd suggest to use sysupdate for this but since it requires root I don't think we want to support it.

[mcassaniti]

Actually I think after taking a look that setting TMPDIR as the workspace here actually resolves the whole issue since systemd-repart should use that temporary directory.

Still working on the PR for this one.

[DaanDeMeyer]

It will use that directory but that's the workspace which is emptied each run. systemd-repart cannot modify existing partitions it will always have to populate the partition from scratch.

[mcassaniti]

systemd-repart cannot modify existing partitions it will always have to populate the partition from scratch.

Of course. I was thinking that reflink based copying could be done but that doesn't really work once you start talking partitions.

[DaanDeMeyer]

I don't see us implementing something like this. As explained in #1809, copies are always going to be required in mkosi's current architecture. We recommend running mkosi on a filesystem that supports reflinks (and optionally subvolumes) for fast image builds.