How to set the security token for the buildkit backend?

[xiaoshao]

I start buildkit with docker(moby/buildkit) as following.

 docker run -d --privileged -p 1234:1234 --name buildkit  moby/buildkit:latest --addr tcp://0.0.0.0:1234 --oci-worker-platform linux/amd64 --oci-worker-platform linux/armhf  

And set the env by export BUILDKIT_HOST=tcp://0.0.0.0:1234 after buildkit is deployed.

But when I build the docker images by

buildctl build --frontend dockerfile.v0  --frontend-opt platform=linux/${PLATFORM} --frontend-opt filename=./${DOCKERFILE_LOCATION} --exporter image --exporter-opt name=${IMAGE}/op_svc_apm:${TAG}-${PLATFORM} --exporter-opt push=true --local dockerfile=.  --local context=.

I got an error as following

error: failed to solve: rpc error: code = Unknown desc = failed to do request: Head https://<my private docker registry>/v2/op_svc_apm/busybox2/manifests/latest: x509: certificate signed by unknown authority

I still got the error after copied the docker.config into the buildkit container.

Could anyone teach me How to fix this problem?

[tonistiigi]

Seems your registry is using a self-signed certificate. You need to add your ca as a trusted root in the buildkit container.

[xiaoshao]

Of course, I have added the ca in the buildkit container,

/etc/docker/certs.d/<my private docker registry host> # ls -al
total 12
drwxr-xr-x    2 root     root          4096 Jun 17 08:48 .
drwxr-xr-x    3 root     root          4096 Jun 17 08:47 ..
-rw-r--r--    1 root     root          1846 Jun 18 00:41 ca.crt

But I still got the error.

buildctl build --frontend dockerfile.v0  --frontend-opt platform=linux/${PLATFORM} --frontend-opt filename=./${DOCKERFILE_LOCATION} --exporter image --exporter-opt name=${IMAGE}/op_svc_apm:${TAG}-${PLATFORM} --exporter-opt push=true --local dockerfile=.  --local context=.
WARN[0000] --exporter <exporter> is deprecated. Please use --output type=<exporter>[,<opt>=<optval>] instead.
WARN[0000] --frontend-opt <opt>=<optval> is deprecated. Please use --opt <opt>=<optval> instead.
[+] Building 40.1s (5/8)
 => [internal] load build definition from ././Dockerfile                                                                                                                                                  0.0s
 => => transferring dockerfile: 32B                                                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                                                         0.1s
 => => transferring context: 2B                                                                                                                                                                           0.0s
 => ERROR [internal] load metadata for <my private docker registry>/op_svc_apm/busybox2:latest                                                                                                                20.0s
 => ERROR [1/4] FROM <my private docker registry>/op_svc_apm/busybox2:latest                                                                                                                                  20.0s
 => => resolve <my private docker registry>/op_svc_apm/busybox2:latest                                                                                                                                        20.0s
 => [internal] load build context                                                                                                                                                                         0.0s
------
 > [internal] load metadata for <my private docker registry>/op_svc_apm/busybox2:latest:
------
------
 > [1/4] FROM <my private docker registry>/op_svc_apm/busybox2:latest:
------
error: failed to solve: rpc error: code = Unknown desc = failed to do request: Head https://<my private docker registry>/v2/op_svc_apm/busybox2/manifests/latest: x509: certificate signed by unknown authority

[tonistiigi]

You need to set them to the system certs. There is no docker running in the container. Eg. https://unix.stackexchange.com/questions/464484/install-self-signed-certificate-to-alpine-linux/464495#464495 should do it.

[xiaoshao]

Thanks @tonistiigi . I could not understand the first very clear (Get a clean environment (This was my first major issue)) .

I just copy the cert to /usr/local/share/ca-certificates and run the command update-ca-certificates .
Then I got the same error.

 update-ca-certificates
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping

[zhujian7]

Seems your registry is using a self-signed certificate. You need to add your ca as a trusted root in the buildkit container.

@tonistiigi
Thanks for your reply. I got Permission denied when I try to append my self-signed certificate to the /etc/ssl/certs/ca-certificates.crt file in the moby/buildkit:rootless container. Is there any guild about how to push the image to a self-signed registry with rootless mode(in a Kubernetes cluster)?

[michaelhixson]

Does buildkitd cache the system certificates somehow? For me it acted as though it did not pick up changes to the installed certs even though other programs (curl) did.

I tried installing the cert manually:

• I was running a moby/buildkit:rootless container
• connect to that container as root with docker exec -it -u root <container-id> sh
• add the cert file to /usr/local/share/ca-certificates
• run update-ca-certificates
• apk add curl for debugging

I kept getting the "certificate signed by unknown authority" error. I could tell the cert was installed correctly because curl -is https://<registry-ip>:<registry-port>/ started accepting the cert even though buildkit kept rejecting it.

What ended up working for me was mounting an already-configured /etc/ssl/certs directory as a volume, so my cert was present in the container at startup time.

• I was running this all in a Kubernetes pod.
• I used an emptyDir volume for the certs.
• I added a moby/buildkit:latest (not rootless) init container mounting that emptyDir volume to /tmp/certs, and mounting my certificate file to /usr/local/share/ca-certificates/my-registry.crt.
• I changed that init container's command to sh -c 'update-ca-certificates; cp -r /etc/ssl/certs/* /tmp/certs'.
• In the non-init moby/buildkit:rootless container, I mounted that same emptyDir volume to /etc/ssl/certs.
• I also had to add login credentials for the registry in the form of echo "{\"auths\": {\"$REGISTRY_HOST_AND_PORT\": {\"auth\": \"$(echo -n "$USERNAME:$PASSWORD" | base64)\"}}}" > /home/user/.docker/config.json.
• Now buildkit can push to the registry successfully.

I feel like there should be an easier way, but at least it worked in the end. Anyway, hopefully this helps someone.

[tonistiigi]

I think the grpc does not support reloading tls keys after files have changed atm.