Command line tool to build packages

[Eeems]

We should have a command line abstractions to handle building packages. Right now you'll just have to know what docker container to run, and how to properly bind in the files and call the appropriate python scripts. Ideally I should just be in a directory with a package file and call toltec-build or something, similar to makepkg. It should then spit out the ipk file(s) in the same directory.

[Bulkin]

I've made a proof of concept for building ipks from a supplied directory and without docker (see Bulkin/toltec@ab54319). The question is, should the finished tool remain in the toltec git repo, or be a separate entity? In my opinion, it would make sense to add the tool to the base toltec-toolchain image and simply call it for each package when building the toltec packages. This would be convenient for app developers, since you get package building for free when using the toltec-toolchain

[Eeems]

My expectation is that docker would still be used, the tool would just handle abstracting that away for you. The tool likely should live in the toolchain repository as it would be part of the toolchain. Thoughts @matteodelabre ?

[matteodelabre]

I’m all for making our build system more standalone. In fact, we talked about moving it to a separate repo before and I still think it’s a good idea since that would enable us to iterate faster on it without compromising on the review requirements of the main repo. However I don’t get the appeal of adding it to the toolchain. The goal of the toolchain repo is only to provide a Docker image with a cross-compiler and a reproducible build environment.

@Bulkin: Your POC looks good! How do you plan on handling packages that depend on Debian packages or on other Toltec packages for building?

[Bulkin]

So I've thought some more on the subject and now believe that it would be best to split the tool into two callable scripts, say toltec-build and toltec-makepkg with the following separation of responsibilities:

toltec-build:

• prepares the docker image (here we would handle the build deps);
• runs toltec-makepkg from the image.

toltec-makepkg:

• expects an environment with all required build dependencies;
• goes through build steps in package;
• creates ipk.

This way, during app development one would typically use toltec-makepkg to create deployable packages, while toltec-build would be called for each package to build the repo (the way builder.make is called now). In other words, for local builds we lose the hard dependency on docker (one could run the toltec-toolchain image with another runner, such as podman, or even use a local toolchain), and nothing much changes for the repo.

It might be a good idea to add toltec-makepkg to the base toolchain image, which would coincidentally be the only reason to put the code in the toltec-toolchain git repo.

My plan is to filter-branch the scripts into a separate git repo named toltec-build. Then after everything works as intended, we could port the toltec package repo to use them. Does this make sense?

P.S. Naming things is hard, all suggestions welcome :)

[matteodelabre]

Seems like a good plan! I’ll look into it after we’re finished with the work on the current stable merge.

[matteodelabre]

On a related note, since we don’t really need all of Docker’s features, we could investigate replacing it with a simpler (chroot+overlayfs)-based approach (see also https://wiki.archlinux.org/title/DeveloperWiki:Building_in_a_clean_chroot).

[matteodelabre]

Just created toltec-dev/build (this repository) with an initial version of the standalone build tool. Usage is toltecmk --work-dir [wdir] --dist-dir [ddir] [rdir], the following directories can be supplied:

• wdir: Working directory into which sources are fetched and builds are run. Can be safely discarded after build is complete. Default: $PWD/build.
• ddir: Directory into which final packages are put. Default: $PWD/dist.
• rdir: Directory where the recipe sources can be found. Should contain a package file. Default: $PWD.

Feedback is welcome! The build tool has been stripped from most of the main-repo-specific logic. The next steps will be to further test this repository, make a release, then update the main repo to use this repo for building.

For now, the Docker separation proposed by @Bulkin is not yet implemented, I’ll open a separate issue for that.