[RFE] Need a good way to add complex Python apps

[dragonpaw]

Current situation

I'm trying to install SatlStack into Flatcar as I'll be using that to schedule reboots across the fleet, etc.

Looking over the different ways to extend Flatcar, it seemed like using the sdk container and adding the salt version from Gentoo's upstream repo would be the most supported way to do this. (I would kinda prefer to do this with a sysext, but that doesn't really seem plausible for a Python app with a bunch of library dependencies and such, plus it's already packaged in Gentoo.)

I came up with this script so far: https://gist.github.com/dragonpaw/1a22cbcc3ae05ed6ff5aade1dc2b8b29

This gets me a lot of the way there, but then I hit a wall with it complaining that there's no version of gnutls available:

there are no ebuilds to satisfy "app-crypt/gnupg[nls]" for /build/amd64-usr/.

(Other attempts end up in some strange loop where it tries to rebuild systemd, which is definiely not anything I want to touch.)

Impact

Right now, I can't find any way to install salt at all. The sysext stuff is cool, but seems to only work for static compiled binaries that can exist out of step with Flatcar releases.

Ideal future situation

Some supported way to add Salt, and other non-static compiled tools like most python apps.

Implementation options

I don't really have a preference for how this is made to work, be it a better way to create a sysext that involves maybe running something insides a chroot with an overlay, or just changing whatever is in the gentoo system to allow emerge to find a solution to the dependencies. Maybe some way to just boot the system, run the install commands, then clear out the machine state with flatcar_reset or the like.

[tormath1]

Hello and thanks for your issue. Adding salt packages and friends to the SDK or the generic image is not ideal as it will increase the size of the image and add Python to the base image which is something we want to avoid as it's not directly related to container runtime, it will increase the attack surface and it will increase the footprint.

The sysext stuff is cool, but seems to only work for static compiled binaries that can exist out of step with Flatcar releases.

This is actually the opposite, sysext images can use dynamic libraries (e.g built-in sysext for Docker, containerd).

We recently added python sysext image (available in the next alpha) - you could try to leverage this image and add the missing pieces for Salt in another image in the sysext-bakery (https://github.com/flatcar/sysext-bakery). It's possible to use multiple sysext images on the same system.

[dragonpaw]

I'm certainly not asking you to add Salt to the base distro of Flatcar. But rather to make it possible for me to add it to my custom version. Because if I can't use Salt to manage it, I can't use the distro. (Changing the fleet management tool we use is a non-starter for me wanting to bring a new distro like Flatcar into the mix.)

I did see the sysext-bakery repo you pointed me at, but as for it being "the opposite" of being for static binaries, the first sentence of the documentation is that sysext-bakery is for "allowing to extend the base OS with custom (static) binaries. " As for it having a python sysext there, the search for 'python' in that repo turns up 0 code hits, and only 1 issue hit: https://github.com/search?q=repo%3Aflatcar%2Fsysext-bakery%20python&type=code

[tormath1]

Yes, that's true sysext-bakery provides images with static binaries.

This is actually the opposite, sysext images can use dynamic libraries (e.g built-in sysext for Docker, containerd).

I was talking about sysext in a generic way, with Flatcar there are three ways to provide sysext images:

• built-in (like Docker or containerd, not statically compiled)
• "official extensions" https://www.flatcar.org/docs/latest/provisioning/sysext/#flatcar-release-extensions like ZFS, Python or Podman (which are not statically compiled)
• from the sysext-bakery (static)

I'm not super familiar with Salt but is that not possible to run the agent from a docker image?

[dragonpaw]

Again, I have the problem that what you say is possible is in contradiction to the published documentation.

Sysext for example, supports 3 file format:

"System extension images may be provided in the following formats:

Plain directories or btrfs subvolumes containing the OS tree

Disk images with a GPT disk label, following the Discoverable Partitions Specification

Disk images lacking a partition table, with a naked Linux file system (e.g. erofs, squashfs or ext4)
"

None of which are a docker containers. So according to the manual for sysext, supplying a docker container is not an option.

(You also mention that docker's binaries are compiled dynamically, but the Flatcar documentation says the opposite: "The Docker releases publish static binaries")

But the goal of this ticket isn't to argue about how Docker is compiled, as it's not in Python and irrelevant really to my question of 'Big, big Python package w/ many C libraries: How?"

Flatcar's own documentation provides this by way of how to customize the image: https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/#making-changes which says the supported way of adding to the image is to clone gentoo's repo and build from that if what you want is already in gentoo's repo. (Which it is and this is exactly what I want.)

As for the link you provide to the "Flatcar Release Extensions", I have no idea what I am supposed to learn from this link other than somehow, by undocumented means, a zfs plugin sysext is available. There's not even a single sentence for how this is accomplished or how I might accomplish a similar trick. If this is another path to customization of Flatcar, it doesn't seem to be documented.

The link you provide to "extra_sysexts.sh", has no comments, documentation, or anything to help me understand what this file does, how I would call it, how customizing it works, what possible values might go into it, etc. At first glance it seems to be gentoo package names, which puts me back at where I started. I am trying to add the gentoo package 'app-admin/salt' but it will not compile because it needs to install over 50 different packages to do so, including an incompatible version of gnutls.

As for if Salt can run inside a docker container, I have no idea. Salt is a very large tool that makes use of libssl, libzmq, libmsgpack, compiled crypto libraries, and a ton of other C libraries, plus 50+ python packages from pip, and it needs to run with full write access to the filesystem (outside of /usr which is of course readonly in Flatcar.) If I try to compile those c libs, I don't know if they'll end up being kernel dependent, depends what's in their ./configure scripts... But if it's running in docker, how would it trigger an OS reboot? (Which as you'll recall was one of my major use cases for having Salt on the system.) It'd also need /etc, /var, /opt, ... all bind mounted so it could configure the system. But the reboot thing seems like a problem. Maybe if systemd's socket was also bindmounted into the container? This is getting weird, but it might work...

[tormath1]

None of which are a docker containers. So according to the manual for sysext, supplying a docker container is not an option.

I supposed I have been unclear in this point. When I talk about "built-in sysext for Docker" I mention this:

$ systemd-sysext status
HIERARCHY EXTENSIONS         SINCE
/opt      none               -
/usr      containerd-flatcar Fri 2024-06-14 11:21:27 UTC
          docker-flatcar
          oem-qemu

By default containerd and docker software are installed in Flatcar as systemd sysext images.

(You also mention that docker's binaries are compiled dynamically, but the Flatcar documentation says the opposite: "The Docker releases publish static binaries")

Correct but the Docker sysext image provided by the sysext-bakery is not the one built in Flatcar (it's an alternative in case you want to use a difference Docker version from the one shipped by Flatcar).

There's not even a single sentence for how this is accomplished or how I might accomplish a similar trick. If this is another path to customization of Flatcar, it doesn't seem to be documented.

This is because it is a low level contribution - there is not a straight-forward way to do this. You could have a look to these pull requests to see how things are done:

• Podman sysext scripts#1964
• Build ZFS sysext with each release scripts#1742
• sysext: Add python sysext scripts#1979

From here I see a few ways:

• you could try to provide a sysext image for Salt
• you could build your own Flatcar image (as you mentioned with this: https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/#making-changes)
• investigate on the Docker support
• (did you have a look to Locksmith to coordinate the reboot?)

[till]

We use Ansible, which may be similar to Salt, for that I use an ansible role to bootstrap pypi and pip.

Maybe pypi could be sysext'd? I tried a few times, but didn't get anywhere. For now we extract pypi to /opt and invoke everything there.

[dragonpaw]

From here I see a few ways:

• you could try to provide a sysext image for Salt
• you could build your own Flatcar image (as you mentioned with this: https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/#making-changes)

We tried doing both of these and failed both times to get it to do what we needed. (This was done and failed before I opened the ticket, and was the reason I opened this ticket.)

In the end we just gave up on using Flatcar. All of us combined and we can't come up with a viable alternative to apt install salt-master or emerge app-admin/salt that works. So we went with another OS.

P.S. I feel like no one took the time to really read my initial request. Because I documented in there quite clearly that I'd already tried both of the listed approaches and why both had failed for us. So to just tell me to try to do what I already tried to do, isn't very helpful to me. This is frustrating and leaves a bad experience. I came here for help, not to be told to go read documentation that I'd clearly already read multiple times and followed until I hit a wall.

[chewi]

I'm sorry this didn't work out for you. I think @tormath1 tried his best to help, but between him lacking knowledge on SaltStack and you lacking knowledge on Flatcar, there was a bit of a disconnect. This is a little more complex than the requests we usually get. I did initially find the thread a little difficult to follow myself. I would also say that the sysext stuff is still relatively new, so we haven't quite established a good method of building them for software with complex dependencies. Being a Gentoo developer, I would love to see better use of Gentoo packages in this area. We did an experiment last year to fill this gap using Gentoo Prefix, but it may still need some finessing. I was new to Flatcar then, and I've only recently returned to it, so I haven't yet had a chance to explore this further. I imagine we'll see more requests like this though, so I'd like to address it before long.

[chewi]

I now remember we discussed this on Matrix. My fault entirely for missing your later replies. Maybe it's too late now, but to answer your question on nls, create /etc/portage/profile/package.use.mask/salt in the SDK with this:

net-libs/gnutls -nls