error when merging multi-platform manifest(s) to custom tls config registry after successfully pushing for individual platforms

[dweomer]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

It looks as if the imagetools.Opt passed to the itpull := imagetools.New(imageopt) line is lacking the necessary RegistryConfig to connect to a private registry signed by a CA that isn't included in the system ca-certificates BUT that individual builders are able to push to without issue (meaning, they are configured properly ... the build pulls from the private build cache registry successfully, honoring the private registry cache importer).

Expected behaviour

Can successfully merge multi-platform manifests for blobs that have already been pushed to a private registry.

Actual behaviour

Cannot successfully merge multi-platform manifests for blobs that have already been pushed to a private registry.

Buildx version

github.com/docker/buildx v0.13.1+dweomer.1 5decc6f

Docker info

Client:
   Version:           25.0.4
   API version:       1.44
   Go version:        go1.21.8
   Git commit:        1a576c5
   Built:             Wed Mar  6 16:32:02 2024
   OS/Arch:           linux/amd64
   Context:           default
  
  Server: Docker Engine - Community
   Engine:
    Version:          26.1.3
    API version:      1.45 (minimum version 1.24)
    Go version:       go1.21.10
    Git commit:       8e96db1
    Built:            Thu May 16 08:33:58 2024
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          v1.7.15
    GitCommit:        926c9586fe4a6236699318391cd44976a98e31f1
   runc:
    Version:          1.1.12
    GitCommit:        v1.1.12-0-g51d5e94
   docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  /usr/bin/docker info
  Client:
   Version:    25.0.4
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.13.1+dweomer.1
      Path:     /home/runner/.docker/cli-plugins/docker-buildx
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 1
   Server Version: 26.1.3
   Storage Driver: overlay2
    Backing Filesystem: xfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: true
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 1
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: 926c9586fe4a6236699318391cd44976a98e31f1
   runc version: v1.1.12-0-g51d5e94
   init version: de40ad0
   Security Options:
    seccomp
     Profile: builtin
   Kernel Version: 5.10.130-118.517.amzn2.x86_64
   Operating System: Alpine Linux v3.19 (containerized)
   OSType: linux
   Architecture: x86_64
   CPUs: 8
   Total Memory: 30.9GiB
   Name: ip-10-10-11-198.us-gov-east-1.compute.internal
   ID: 941f2083-c5f9-4f79-8d28-fb49661dfb6c
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Registry Mirrors:
    https://dhub.cache.svc/
   Live Restore Enabled: false
   Product License: Community Engine

Builders list

Name:          builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e
  Driver:        kubernetes
  Last Activity: 2024-05-25 13:46:09 +0000 UTC
  Nodes:
  Name:                  builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e0
  Endpoint:              kubernetes:///builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e?deployment=buildkit-495d6f30-f49f-491d-a811-0cf9049bccc6-8tfds&kubeconfig=
  Driver Options:        nodeselector="category=build" tolerations="key=category,value=build"
  Status:                running
  BuildKit daemon flags: --allow-insecure-entitlement=network.host
  BuildKit version:      v0.13.2
  Platforms:             linux/amd64*, linux/amd64/v2*, linux/amd64/v3*, linux/amd64/v4*, linux/386*, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
  Labels:
   org.mobyproject.buildkit.worker.executor:         oci
   org.mobyproject.buildkit.worker.hostname:         builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e0-6df88cfdc6-88rrb
   org.mobyproject.buildkit.worker.network:          host
   org.mobyproject.buildkit.worker.oci.process-mode: sandbox
   org.mobyproject.buildkit.worker.selinux.enabled:  false
   org.mobyproject.buildkit.worker.snapshotter:      overlayfs
  GC Policy rule#0:
   All:           false
   Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
   Keep Duration: 48h0m0s
   Keep Bytes:    488.3MiB
  GC Policy rule#1:
   All:           false
   Keep Duration: 1440h0m0s
   Keep Bytes:    46.57GiB
  GC Policy rule#2:
   All:        false
   Keep Bytes: 46.57GiB
  GC Policy rule#3:
   All:        true
   Keep Bytes: 46.57GiB
  Name:                  builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e1
  Endpoint:              kubernetes:///builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e?deployment=buildkit-9e169c6c-af0d-46c4-9ad3-1589f6a15580-5lp28&kubeconfig=
  Driver Options:        nodeselector="category=build-arm64" tolerations="key=category,value=build-arm64"
  Status:                running
  BuildKit daemon flags: --allow-insecure-entitlement=network.host
  BuildKit version:      v0.13.2
  Platforms:             linux/arm/v6*, linux/arm/v7*, linux/arm64*
  Labels:
   org.mobyproject.buildkit.worker.executor:         oci
   org.mobyproject.buildkit.worker.hostname:         builder-0be5ebbb-1707-47bc-bd90-13edc91cfb1e1-cb9c654df-kxqgr
   org.mobyproject.buildkit.worker.network:          host
   org.mobyproject.buildkit.worker.oci.process-mode: sandbox
   org.mobyproject.buildkit.worker.selinux.enabled:  false
   org.mobyproject.buildkit.worker.snapshotter:      overlayfs
  GC Policy rule#0:
   All:           false
   Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
   Keep Duration: 48h0m0s
   Keep Bytes:    488.3MiB
  GC Policy rule#1:
   All:           false
   Keep Duration: 1440h0m0s
   Keep Bytes:    46.57GiB
  GC Policy rule#2:
   All:        false
   Keep Bytes: 46.57GiB
  GC Policy rule#3:
   All:        true
   Keep Bytes: 46.57GiB

Configuration

FROM library/alpine:edge
RUN echo 'unable to share this but the same dockerfile merges just fine to ghcr.io'

Build logs

#26 exporting to image
#26 ...
#27 exporting to image
#27 exporting layers
#27 ...
#26 exporting to image
#26 exporting layers 65.5s done
#26 exporting manifest sha256:d9be2cdb45c5b07b54691e153ad5b6b4c8d527356500323f9fea81df300876c5 done
#26 exporting config sha256:17675bb9b8dc515066bc0f326b2d548dfe1232579f588a1cbdbf5c45a7f726cd done
#26 exporting attestation manifest sha256:8c81834da520a243b0fd108537c970d94c4bee5e270a6bfcd74b9c8a38854e5f 0.0s done
#26 exporting manifest list sha256:dc6732014a6873697cadcc9531c31b7504193fbecac1411ea3491d8118152bcb done
#26 pushing layers
#26 pushing layers 3.8s done
#26 pushing manifest for build.cache.svc/my-project/my-image
#26 pushing manifest for build.cache.svc/my-project/my-image 0.0s done
#26 DONE 69.4s
#28 exporting cache to registry
#28 preparing build cache for export
#28 writing layer sha256:08b1720df82a0beee132289941ac9ee2eba74a7d2ad637c1a8352366d751fb25 done
#28 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 done
#28 writing layer sha256:561cb69653d56a9725be56e02128e4e96fb434a8b4b4decf2bdeb479a225feaf done
#28 writing layer sha256:8f665685b215c7daf9164545f1bbdd74d800af77d0d267db31fe0345c0c8fb8b done
#28 writing layer sha256:9361d72813976e1175ddb2fbce2e5f0ab01e71a419990d64e71bc36946edd884 done
#28 writing layer sha256:96ad531c39c935bc6319f19f3be8f9f4a6faa15ded833ad2bd50a95a0d95e8d2 done
#28 writing layer sha256:e5fca6c395a62ec277102af9e5283f6edb43b3e4f20f798e3ce7e425be226ba6 done
#28 writing layer sha256:f56be85fc22e46face30e2c3de3f7fe7c15f8fd7c4e5add29d7f64b87abdaa09 done
#28 writing layer sha256:fc07f0dda8ec1c1acc98ab6a4673371611db7184cff56ddef0eba11523eec347 done
#28 writing config sha256:5bf508bda394326c3229d6ad06bcb6bded9357713a60be9a5056503b68adbadf 0.0s done
#28 writing cache manifest sha256:e681f494875749ceb3083097acf67ee72e298cc01dce5ab63e5f856b65cbf12c
#28 preparing build cache for export 0.1s done
#28 writing cache manifest sha256:e681f494875749ceb3083097acf67ee72e298cc01dce5ab63e5f856b65cbf12c 0.0s done
#28 DONE 0.1s
#27 exporting to image
#27 exporting layers 68.0s done
#27 exporting manifest sha256:c3092a12a16f9d5411701e95592b1f0d0d64b24ff810727cf911128403848f11 done
#27 exporting config sha256:459fb84f04c080c7a977c605b777e903e4a135002b442ede68aed725320f5880 done
#27 exporting attestation manifest sha256:c13a9929f49c119f9dccbeeeb763a84548efa08dea86a7546ad9a128dbc5e9c5 0.0s done
#27 exporting manifest list sha256:fc95710499e7bb88684294ec76c284fa9c73444468793654a5c68ecb3b059397 done
#27 pushing layers
#27 pushing layers 3.7s done
#27 pushing manifest for build.cache.svc/my-project/my-image
#27 pushing manifest for build.cache.svc/my-project/my-image 0.0s done
#27 DONE 71.7s
#29 exporting cache to registry
#29 preparing build cache for export
#29 writing layer sha256:4cf6a83c0e2af3c780abcda02cc33f9e812fdcb40b610ed1838281cc9ab94ec8 done
#29 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 done
#29 writing layer sha256:5a63f40ac9bbdfab87854860e46116e14c81556e0d159437bcbd13ec83848687
#29 preparing build cache for export 0.1s done
#29 writing layer sha256:5a63f40ac9bbdfab87854860e46116e14c81556e0d159437bcbd13ec83848687 done
#29 writing layer sha256:683339ce8d6b9be2ca150a8de67b895e20ea5594b91d3911c95b0b8fea3e314c done
#29 writing layer sha256:686172e40c38722891b4004f55f6447548c8367968ac523a612591e0d92f9db3 done
#29 writing layer sha256:c41833b44d910632b415cd89a9cdaa4d62c9725dc56c99a7ddadafd6719960f9 done
#29 writing layer sha256:e83c0d77c542c0ae16eda4f948bdc6e84b0a82b8a00068b7eeb5a5a743b1b453 done
#29 writing layer sha256:ed43d91b02ce995d68736bc3af861c28500f6109fcb8d62179c71ffa023ce97a done
#29 writing layer sha256:fc1eefa94020698f74056fc3449798c2319f23cb42221d278064fa8f8ea616c0 done
#29 writing config sha256:95ee56b834bf8aa0dde7ef40d4fe16146f00da17d3c14ca69fabb7aafe8f9e87 0.0s done
#29 writing cache manifest sha256:777b29ca996df891e166c85a82232c6da4b94c19470a3d5ca32c0641144ede04 0.0s done
#29 DONE 0.1s
#30 merging manifest list build.cache.svc/my-project/my-image:my-tag,build.cache.svc/my-project/my-image:sha-cc220b522f58843a818603b89cf6195fd4b30643,build.cache.svc/my-project/my-image:latest
#30 ERROR: httpReadSeeker: failed open: failed to do request: Get "https://build.cache.svc/v2/my-project/my-image/manifests/sha256:fc95710499e7bb88684294ec76c284fa9c73444468793654a5c68ecb3b059397": tls: failed to verify certificate: x509: certificate signed by unknown authority
------
 > merging manifest list build.cache.svc/my-project/my-image:my-tag,build.cache.svc/my-project/my-image:sha-cc220b522f58843a818603b89cf6195fd4b30643,build.cache.svc/my-project/my-image:latest:
------

Additional info

This is driven via github actions on a private runner, leveraging:

• https://github.com/marketplace/actions/docker-metadata-action
• https://github.com/marketplace/actions/docker-setup-buildx
• https://github.com/marketplace/actions/build-and-push-docker-images
• https://github.com/dweomer/buildx/releases/tag/v0.13.1%2Bdweomer.1

See also:

• kubernetes: fix configmap volumes #2462

[dweomer]

So, updating the system ca trust got me further, i.e.:

# runner startup tweaks
sudo ln -vs /etc/docker/certs.d/build.cache.svc/ca.crt /usr/local/share/ca-certificates/build-cache-svc.crt
sudo update-ca-certificates

But then I see this in the access logs for build.cache.svc:

10.42.165.22 - - [25/May/2024:16:14:08 +0000] "PUT /v2/my-project/my-image/manifests/my-tag HTTP/1.1" 201 0 "" "buildkit/v0.13"
2024-05-25T16:14:08.762990483Z time="2024-05-25T16:14:08.7627864Z" level=info msg="response completed" go.version=go1.20.8 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=build.cache.svc http.request.id=f0f62fae-b358-4b4f-af1c-64d8f2ddbdc8 http.request.method=PUT http.request.remoteaddr="10.42.165.22:58088" http.request.uri="/v2/my-project/my-image/manifests/my-tag" http.request.useragent="buildkit/v0.13" http.response.duration=7.41057ms http.response.status=201 http.response.written=0 
2024/05/25 16:14:08 http: TLS handshake error from 10.42.53.0:63128: tls: client didn't provide a certificate
2024/05/25 16:14:08 http: TLS handshake error from 10.42.53.0:45160: EOF

Progress! Now the issue is the client (buildx imagetools, I believe) is neglecting to send a client certificate when pushing to the private build cache registry. So, I tried these additional tweaks at runner startup:

# match default expectations of https://github.com/docker/setup-buildx-action
# sans a DOCKER_CONFIG env override (which fails due to attempted writes on a read-only volumeMount)
ln -vs /etc/docker/certs.d ~/.docker/
# match the translated buildkitd.toml registry ca/keypair entries shipped to buildkitd backends
sudo mkdir -p /etc/buildkit
sudo ln -vs /etc/docker/certs.d /etc/buildkit/certs

... but still no joy.

AFAIK I have correctly followed the mTLS setup for my build-cache registry, i.e.:

runner@ip-10-10-11-198:~$ ls -alF /etc/docker/certs.d/build.cache.svc/
total 0
dr-xr-x--- 3 root docker 140 May 25 16:14 ./
drwxr-xr-x 7 root root   122 May 25 16:15 ../
drwxr-xr-x 2 root root   100 May 25 16:14 ..2024_05_25_16_14_57.1642649139/
lrwxrwxrwx 1 root root    32 May 25 16:14 ..data -> ..2024_05_25_16_14_57.1642649139/
lrwxrwxrwx 1 root root    13 May 25 16:14 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root    18 May 25 16:14 client.cert -> ..data/client.cert
lrwxrwxrwx 1 root root    17 May 25 16:14 client.key -> ..data/client.key

[dweomer]

hi @tonistiigi, i do not understand this edit:

our build.cache.svc registry is secured via mTLS (with some dummy basic auth) which buildkitd backend(s) push to without issue but with the same config present on the client isn't recognized by buildx imagetools (some call into that go package, i forget) and so it doesnt expect the custom ca nor does it use the client certificate when negotiating an https connection. am i missing something? the "insecure registry" designation is a category error, from my perspective.

[dweomer]

was digging into this further yesterday (because i am unreasonably angry that the awesome buildx imagetools fails like this) and it looks like a problem with the containerd resolver that imagetools is relying on (i even tried an /etc/containerd/certs.d/_default/hosts.toml with no luck). something truncated in the setup, likely a gap between what ctr adds and imagetools expects.

also, docker push to this mTLS secured registry works fine via the tlscacert, tlscert, and tlskey flags to docker but these are not picked up by buildx imagetools.

likely related issues:

• buildkitd registry-auth-tlscontext flag not support #2379
• Self signed certificate cannot be authenticated #2008