feat: Allow adding stages to run builds for things like compiled programs

[gmpinder]

I've been thinking of this idea for a while now and I've heard from one of our users that they would like to do something similar. So now that we've proven that multi-stage builds will work for us, we can take advantage of that to allow users to create their own stages. This would be useful for compiling software that might require a specific set of dependencies that you don't want in your final image.

Spec

For creating a stage, I thought that the best way to represent a set of stages was to add a stages property in the top level of the file. Here you have a list of stages with the required properties of name, image, and modules. The user would then be able to add modules to call from within the stage. In the example below, I'm using the containerfile module to build bluebuild using cargo on the rust image. Then in the main modules, I'm using COPY to copy the bin into the final image without having to install all of the cargo dependencies in my final image.

name: jp-desktop
base-image: ghcr.io/ublue-os/bazzite-nvidia
image-version: stable
description: The image of Wunker OS for JP's desktop.
blue-build-tag: main-installer
stages:
  - name: blue-build
    image: rust
    modules:
      - type: containerfile
        snippets:
          - |
            RUN cargo install blue-build --features nightly --target x86_64-unknown-linux-gnu \
              && mkdir -p /out/ \
              && mv $CARGO_HOME/bin/bluebuild /out/bluebuild
modules:
  - type: containerfile
    snippets:
      - COPY --from=blue-build /out/bluebuild /usr/bin/bluebuild

The resulting Containerfile would end up looking something like this:

# ...
FROM rust as blue-build
RUN cargo install blue-build --features nightly --target x86_64-unknown-linux-gnu \
              && mkdir -p /out/ \
              && mv $CARGO_HOME/bin/bluebuild /out/bluebuild

FROM ghcr.io/ublue-os/bazzite-nvidia

# ...
COPY --from=blue-build /out/bluebuild /usr/bin/bluebuild

Adding support for from-file: would also make this extremely modular and allow for easy reuse. It can follow the same patter where it could be a single stage:

name: blue-build
image: rust
modules:
  - type: containerfile
    snippets:
      - |
        RUN cargo install blue-build --features nightly --target x86_64-unknown-linux-gnu \
          && mkdir -p /out/ \
          && mv $CARGO_HOME/bin/bluebuild /out/bluebuild

Or it can hold multiple ones

stages:
  - name: blue-build
    image: rust
    modules:
      - type: containerfile
        snippets:
          - |
            RUN cargo install blue-build --features nightly --target x86_64-unknown-linux-gnu \
              && mkdir -p /out/ \
              && mv $CARGO_HOME/bin/bluebuild /out/bluebuild
  - name: cargo-release
    image: rust
    modules:
      - type: containerfile
        snippets:
          - |
            RUN cargo install cargo-release --locked --target x86_64-unknown-linux-gnu \
              && mkdir -p /out/ \
              && mv $CARGO_HOME/bin/cargo-release /out/cargo-release

[LelouBil]

I find this perfectly usable, it just needs to allow the same inclusion of other .yml files that main modules do.

And yeah it's really helpful to include software not packaged for fedora in a universal blue system, it auto updated on each build and a bash script to build a software is way simpler than figuring out how to package it as a rpm

[xynydev]

This sems alright, I'm just kind of hesitant about adding more to the top-level recipe namespace, especially as base image verification etc. will also require such additions.

I had initially thought that this could be implemented as a module or a flag to the containerfile module, that would allow just inserting into the containerfile before the main FROM instruction. I'm not sure what the usefulness of being able to use modules in stages is, as most stages would probably be just doing RUN or COPY.

[gmpinder]

This sems alright, I'm just kind of hesitant about adding more to the top-level recipe namespace, especially as base image verification etc. will also require such additions.

I had initially thought that this could be implemented as a module or a flag to the containerfile module, that would allow just inserting into the containerfile before the main FROM instruction. I'm not sure what the usefulness of being able to use modules in stages is, as most stages would probably be just doing RUN or COPY.

Having a separate stages section will help to keep the logic for managing modules cleaner and keep a very distinct separation of duties for these different functions. Allowing the use of modules would then allow reuse of existing configs to perform steps like the scripts module or files module and keep the users recipe file cleaner and easier to understand.

This also has the benefit of conveying to the user that modules that happen in a stage are distinctly separate from their final image. Reuse of stages would also allow greater configurability for larger projects like secureblue.

We could also add a fromor from-image property for the files module to make it easier to copy from stages or other images without the need to explicitly define containerfile snippets.

[xynydev]

Just dropping this enough. Kind of funny that Vib just introduced stages. https://docs.vanillaos.org/vib/en/recipe-structure

[gmpinder]

Just dropping this enough. Kind of funny that Vib just introduced stages. https://docs.vanillaos.org/vib/en/recipe-structure

Yeah similar to that!