buildkitd is not available for containerd runtime in colima 0.30.x

[spkane]

• Initial Issue:

$ colima start --cpu 8 --memory 8 --disk 100 --runtime containerd --with-kubernetes
$ kubectl config set current-context --namespace=default colima

$ nerdctl build --namespace k8s.io --platform linux/amd64 -t example/example:local -f ./Dockerfile .
ERRO[0000] error: failed to list workers: Unavailable: connection error: desc = "transport: error while dialing: dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory" 
FATA[0000] `buildctl` needs to be installed and `buildkitd` needs to be running, see https://github.com/moby/buildkit: exit status 1 
FATA[0000] exit status 1                                
make: *** [nerdctl-local-arch-build] Error 1

• Workaround:

$ colima ssh
colima:/Users/spkane/dev/example/example$ sudo buildkitd &
INFO[2022-01-14T18:47:29Z] auto snapshotter: using overlayfs
...
colima:/Users/spkane/dev/example/example$ exit

nerdctl build --namespace k8s.io --platform linux/amd64 -t example/example:local -f ./Dockerfile .
[+] Building 79.2s (19/19) FINISHED                                                                      
 => [internal] load build definition from Dockerfile                                                0.1s
...

If this is intentional, it would be nice to include a buildkitd init file that could easily be enabled for folks who need it.

[spkane]

You can use colima ssh to connect to the VM and then using sudo vi to create the file /etc/init.d/buildkitd like so:

#!/sbin/openrc-run
supervisor=supervise-daemon

name="Buildkit Daemon"
description="Persistent process that manages buildkit"
description_reload="Reload configuration without exiting"

command="${BUILDKITD_BINARY:-/usr/local/bin/buildkitd}"
command_args="${BUILDKITD_OPTS}"
BUILDKITD_LOGFILE="${BUILDKITD_LOGFILE:-/var/log/${RC_SVCNAME}.log}"
BUILDKITD_ERRFILE="${BUILDKITD_ERRFILE:-${BUILDKITD_LOGFILE}}"
BUILDKITD_OUTFILE="${BUILDKITD_OUTFILE:-${BUILDKITD_LOGFILE}}"
supervise_daemon_args="--stderr \"${BUILDKITD_ERRFILE}\" --stdout \"${BUILDKITD_OUTFILE}\""

extra_started_commands="reload"

rc_ulimit="${BUILDKITD_ULIMIT:--c unlimited -n 1048576 -u unlimited}"

retry="${BUILDKITD_RETRY:-TERM/60/KILL/10}"

depend() {
        need sysfs cgroups
        after iptables ip6tables
}

start_pre() {
        checkpath -f -m 0644 -o root:root "$BUILDKITD_ERRFILE" "$BUILDKITD_OUTFILE"
}

reload() {
        ebegin "Reloading configuration"
        $supervisor $RC_SVCNAME --signal HUP
        eend $?
}

And then add that startup script to the default runlevel with:

colima ssh sudo rc-update add buildkitd default

At the moment, I am not seeing it autostart when the VM comes up (this might be an error in my startup script which I mostly borrowed from the docker version.

However, this works fine to get it started:

$ colima ssh sudo rc-service buildkitd restart

[abiosoft]

Wow, this is a major bug. I will look at adding some integration tests to prevent this in the future.
My mistake was focusing more on docker testing.

Thanks for spotting this, will work on a fix and release a new version soonest.

[abiosoft]

At the moment, I am not seeing it autostart when the VM comes up (this might be an error in my startup script which I mostly borrowed from the docker version.

Actually this is preferred. Colima starts the required services after startup.

[abiosoft]

@spkane it might take a while for the next release to be available on brew, but can you test it now by installing with brew install --head colima. It should be fixed now.

[abiosoft]

Release created https://github.com/abiosoft/colima/releases/tag/v0.3.2

[spkane]

@abiosoft The new release appears to be working.

I did get a DNS related error on my first test, but simply re-running the command seems to have fixed it up.

For the record:

$ nerdctl build --namespace k8s.io --platform linux/amd64 -t test/test:local -f ./Dockerfile .
[+] Building 0.2s (4/4) FINISHED
...
error: failed to solve: alpine:latest: failed to do request: Head "https://registry-1.docker.io/v2/library/alpine/manifests/latest": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:45220->[::1]:53: read: connection refused
FATA[0000] unrecognized image format
FATA[0000] exit status 1

[abiosoft]

Hmmm, looks like I will include image pulling in the integration tests. I was trying to avoid getting rate limited by DockerHub.

[spkane]

I teach a bunch of classes on containers and k8s and I have seen students have a similar one-time DNS issue like this, and the one common factor that I can think of is that Docker and Colima both use and underlying alpine container and, of course, have buildkit functionality. I am guessing that this is something alpine related though. Maybe a service that isn't started until it is used, or some cache that needs to be warmed? Not really sure, but it wouldn't suprise me if it was related to Alpine itself.

A very simple google search suggests that there is indeed some "known" issues with DNS in Alpine.