composefs should probably default to single squashed layer

[cgwalters]

Today the c/storage composefs backend generates a composefs blob per layer and unions them via overlayfs.

In contrast in ostree, we always generate a single final flattened filesystem tree.

For any container image where we want to verify its integrity, I think it's just far simpler logistically to sign a single composefs digest of the flattened filesystem tree (xref composefs/composefs#294 ).

Even apart from signatures, I think it will probably just be more efficient to do things this way in most cases.

That said, we might investigate a hybrid approach where a layer chain that is referenced a large percentage of times is still split.

(Simple example: User runs 150 different microservice apps all using a canonical base image; if we flatten then we get 150x base image metadata cost; in the non-integrity case we could still dynamically union the two stacks w/overlayfs, with a composefs for each. But, taking this out to the integrity case gets messier)

[rhatdan]

@giuseppe @alexlarsson PTAL

[cgwalters]

Tangentially related to this...if I'm understanding things right, we today create a mount in the global mount namespace for each layer because that's the main way for us to ensure that we can share layers across multiple containers.

But if we did, this, the need to manipulate the global mount namespace could drop to basically nil by default, which would be quite nice. Then https://github.com/containers/kubensmnt wouldn't have a reason to exist at all...

[giuseppe]

we don't leave the EROFS mount per layer around today. We create them temporarily, then mount overlay on top of them and unmount the EROFS mounts.

overlay grabs a reference to the mount point, and it is used internally in the kernel, but it is not accessible anymore from the VFS. That is why we see only one overlay mount even if EROFS is used, because these mounts are not reachable anymore from the host.

[cgwalters]

cc composefs/composefs#294 which is touching on this one too.

---

Backing up more broadly I do want to say that I strongly think composefs should become the "technical heart" behind how we store container images and the booted host (and probably beyond that, maybe e.g. NixOS will pick it up too).

But still today (for mostly good/understandable reasons) we've basically just "glued" composefs alongside ostree and c/storage. The good reason for this is that rewriting things is hard. There's just a lot of Stuff in c/storage (how we represent layers, images, etc.) that's all part of the overlay driver. Similarly in ostree...a lot of Stuff around how we represent "deployments" and the bootloader entries and metadata about the OS (and fsck) that's just wired to ostree's legacy representation, not composefs.

This all said, I do think it really does make sense to progress on making composefs the core, and less of something we just kind of awkwardly glue onto the side.