podman buildx vs docker buildx in devcontainers, behavior inconsistent, adding features is impossible due to not found proxy

[jankap]

• VSCode Version: 1.101.2
• Local OS Version: Windows 11
• Remote OS Version: WSL2?
• Remote Extension/Connection Type: Dev Containers/WSL/Remote - Tunnels/VS Code Server

I'm highly confused. I use Podman Desktop on Windows 11 with properly set Proxy variables. I'm behind a corporate ZScaler proxy.

podman run -it --rm curlimages/curl -v https://www.google.com works and can push through the proxy.

Assuming this simple devcontainer.json (clone https://github.com/jankap/devcontainer-tests/tree/main for the full example)

{
	"name": "Debian",
	"image": "mcr.microsoft.com/devcontainers/base:bullseye",
	"features": {
		"ghcr.io/devcontainers/features/git:1": {}
	}
}

VSCode and devcontainer show different behaviors.

Scenario 1: "Clone Repository in Volume" in VSCode

1. Clone repo in named volume
2. bootstrap container is created, docker buildx is called (although I'm using podman...), and the build failed because of non-forwarded proxy variables (see e.g. Env Vars (proxy) specified in devcontainer.json don't propagate to containers used to build/add advanced features #6464, Proxy Settings devcontainers/cli#927, and especially Error during build using corporate proxy moby/buildkit#4725)

Error adding the user features:

[12257 ms] --- Fetching User Features ----
[12257 ms] * Fetching feature: git_0_oci
[12257 ms] Fetching from OCI
[12261 ms] blob url: https://ghcr.io/v2/devcontainers/features/git/blobs/sha256:fd557a670d0bca81c4a6f28dd94d36cc3dc6260234b1246ce21b256511bd9ed0
[12261 ms] [httpOci] Applying cachedAuthHeader for registry ghcr.io...
[12715 ms] [httpOci] 200 (Cached): https://ghcr.io/v2/devcontainers/features/git/blobs/sha256:fd557a670d0bca81c4a6f28dd94d36cc3dc6260234b1246ce21b256511bd9ed0
[12717 ms] omitDuringExtraction: '
[12717 ms] Files to omit: ''
[12720 ms] Testing './'(Directory)
[12721 ms] Testing './NOTES.md'(File)
[12722 ms] Testing './README.md'(File)
[12722 ms] Testing './devcontainer-feature.json'(File)
[12722 ms] Testing './install.sh'(File)
[12725 ms] Files extracted from blob: ./NOTES.md, ./README.md, ./devcontainer-feature.json, ./install.sh
[12726 ms] * Fetched feature: git_0_oci version 1.3.4
[12730 ms] Start: Run: docker buildx build --load --build-context dev_containers_feature_content_source=/tmp/devcontainercli-root/container-features/0.77.0-1751027457612 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/base:bullseye --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -f /tmp/devcontainercli-root/container-features/0.77.0-1751027457612/Dockerfile.extended -t vsc-devcontainer-tests-8429b381ca25d45d7fac07afd6273d94bfa5e5d7560d05e8ba216ed51ee45284-features /tmp/devcontainercli-root/empty-folder
[+] Building 0.0s (0/1)                                                                      docker-container:default
[+] Building 64.6s (3/3) FINISHED                                                            docker-container:default
 => [internal] booting buildkit                                                                                 34.3s
 => => pulling image moby/buildkit:buildx-stable-1                                                              32.8s
 => => creating container buildx_buildkit_default                                                                1.5s
 => [internal] load build definition from Dockerfile.extended                                                    0.1s
 => => transferring dockerfile: 2.28kB                                                                           0.0s
 => ERROR resolve image config for docker-image://docker.io/docker/dockerfile:1.4                               30.0s
------
 > resolve image config for docker-image://docker.io/docker/dockerfile:1.4:
------
Dockerfile.extended:1
--------------------
   1 | >>> # syntax=docker/dockerfile:1.4
   2 |     ARG _DEV_CONTAINERS_BASE_IMAGE=placeholder
   3 |     
--------------------
ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: failed to resolve source metadata for docker.io/docker/dockerfile:1.4: failed to do request: Head "https://registry-1.docker.io/v2/docker/dockerfile/manifests/1.4": dial tcp 98.85.153.80:443: i/o timeout

As already reported here: #11032

Key issue is that the proxy is not set when calling buildkit.

What confuses me more is, that always docker buildx is used, instead of podman buildx, although I set VSCode to use podman: "dev.containers.dockerPath": "podman"

More detailed log:

full_log.txt

Scenario 2: Use devcontainer cli

devcontainer build --no-cache
[2025-06-27T12:54:57.845Z] @devcontainers/cli 0.77.0. Node.js v22.15.1. win32 10.0.22631 x64.
[2025-06-27T12:54:58.679Z] Resolving Feature dependencies for 'ghcr.io/devcontainers/features/git:1'...
[2025-06-27T12:54:59.557Z] Soft-dependency 'ghcr.io/devcontainers/features/common-utils' is not required.  Removing from installation order...
[2025-06-27T12:54:59.905Z] Files to omit: ''
[2025-06-27T12:54:59.924Z] Start: Run: podman buildx build --load --build-context dev_containers_feature_content_source=C:\Users\<user>\AppData\Local\Temp\devcontainercli\container-features\0.77.0-1751028898677 --no-cache --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/base:bullseye --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -f C:\Users\<user>\AppData\Local\Temp\devcontainercli\container-features\0.77.0-1751028898677\Dockerfile.extended -t vsc-devcontainer-tests-01e12d5d47ae16210f2220c734de0f4cac83fef5384a8953e00b860a51d49220-features C:\Users\<user>\AppData\Local\Temp\devcontainercli\empty-folder
[2025-06-27T12:55:00.164Z] [1/2] STEP 1/4: FROM mcr.microsoft.com/devcontainers/base:bullseye AS dev_containers_feature_content_normalize
[2025-06-27T12:55:00.683Z] [1/2] STEP 2/4: USER root
[2025-06-27T12:55:00.734Z] --> 82dda63dd474
[2025-06-27T12:55:00.744Z] [1/2] STEP 3/4: COPY --from=dev_containers_feature_content_source devcontainer-features.builtin.env /tmp/build-features/
[2025-06-27T12:55:02.011Z] --> ca3af5e51417
[2025-06-27T12:55:02.031Z] [1/2] STEP 4/4: RUN chmod -R 0755 /tmp/build-features/
[2025-06-27T12:55:03.130Z] --> 29fa63411c3d
[2025-06-27T12:55:03.142Z] [2/2] STEP 1/9: FROM mcr.microsoft.com/devcontainers/base:bullseye AS dev_containers_target_stage
[2025-06-27T12:55:03.650Z] [2/2] STEP 2/9: USER root
[2025-06-27T12:55:03.698Z] --> d13847e4bcd7
[2025-06-27T12:55:03.710Z] [2/2] STEP 3/9: RUN mkdir -p /tmp/dev-container-features
[2025-06-27T12:55:05.115Z] --> 668e66e47d0b
[2025-06-27T12:55:05.127Z] [2/2] STEP 4/9: COPY --from=dev_containers_feature_content_normalize /tmp/build-features/ /tmp/dev-container-features
[2025-06-27T12:55:06.148Z] --> 93bf164e6660
[2025-06-27T12:55:06.165Z] [2/2] STEP 5/9: RUN echo "_CONTAINER_USER_HOME=$( (command -v getent >/dev/null 2>&1 && getent passwd 'root' || grep -E '^root|^[^:]*:[^:]*:root:' /etc/passwd || true) | cut -d: -f6)" >> /tmp/dev-container-features/devcontainer-features.builtin.env && echo "_REMOTE_USER_HOME=$( (command -v getent >/dev/null 2>&1 && getent passwd 'vscode' || grep -E '^vscode|^[^:]*:[^:]*:vscode:' /etc/passwd || true) | cut -d: -f6)" >> /tmp/dev-container-features/devcontainer-features.builtin.env
[2025-06-27T12:55:07.114Z] --> 4196569c6500
[2025-06-27T12:55:07.127Z] [2/2] STEP 6/9: RUN --mount=type=bind,from=dev_containers_feature_content_source,source=git_0,target=/tmp/build-features-src/git_0     cp -ar /tmp/build-features-src/git_0 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/git_0  && cd /tmp/dev-container-features/git_0  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/git_0
[2025-06-27T12:55:07.480Z] ===========================================================================
Feature       : Git (from source)
Description   : Install an up-to-date version of Git, built from source as needed. Useful for when you want the latest and greatest features. Auto-detects latest stable version and installs needed dependencies.
Id            : ghcr.io/devcontainers/features/git
Version       : 1.3.4
Documentation : https://github.com/devcontainers/features/tree/main/src/git
Options       :
    VERSION="os-provided"
    PPA="true"
===========================================================================
[2025-06-27T12:55:07.521Z] Detected existing system install: git version 2.49.0
[2025-06-27T12:55:08.244Z] --> e2803884d69f
[2025-06-27T12:55:08.256Z] [2/2] STEP 7/9: ARG _DEV_CONTAINERS_IMAGE_USER=root
[2025-06-27T12:55:08.302Z] --> 845579cf36a8
[2025-06-27T12:55:08.315Z] [2/2] STEP 8/9: USER $_DEV_CONTAINERS_IMAGE_USER
[2025-06-27T12:55:08.363Z] --> 459cbbd4b96c
[2025-06-27T12:55:08.373Z] [2/2] STEP 9/9: LABEL devcontainer.metadata="[ {\"id\":\"ghcr.io/devcontainers/features/common-utils:2\"}, {\"id\":\"ghcr.io/devcontainers/features/git:1\",\"customizations\":{\"vscode\":{\"settings\":{\"github.copilot.chat.codeGeneration.instructions\":[{\"text\":\"This dev container includes an up-to-date version of Git, built from source as needed, pre-installed and available on the `PATH`.\"}]}}}}, {\"remoteUser\":\"vscode\"}, {\"id\":\"ghcr.io/devcontainers/features/git:1\",\"customizations\":{\"vscode\":{\"settings\":{\"github.copilot.chat.codeGeneration.instructions\":[{\"text\":\"This dev container includes an up-to-date version of Git, built from source as needed, pre-installed and available on the `PATH`.\"}]}}}} ]"
[2/2] COMMIT vsc-devcontainer-tests-01e12d5d47ae16210f2220c734de0f4cac83fef5384a8953e00b860a51d49220-features
[2025-06-27T12:55:08.422Z] --> 4685451021ef
[Warning] one or more build args were not consumed: [_DEV_CONTAINERS_FEATURE_CONTENT_SOURCE]
[2025-06-27T12:55:08.423Z] Successfully tagged localhost/vsc-devcontainer-tests-01e12d5d47ae16210f2220c734de0f4cac83fef5384a8953e00b860a51d49220-features:latest
[2025-06-27T12:55:08.554Z] 4685451021ef66dce9af07ae1a6ab23bfa6b8f3d704da14b697e7811c2c6a4e5
{"outcome":"success","imageName":["vsc-devcontainer-tests-01e12d5d47ae16210f2220c734de0f4cac83fef5384a8953e00b860a51d49220-features"]}

I.e., using the devcontainer cli just works.

Key question

Why is VSCode using docker buildx, whereas the devcontainer CLI is (correctly) using podman buildx? And how can I change that?

[chrmarti]

Could you append the full log from when it fails?

[jankap]

Could you append the full log from when it fails?

@chrmarti

It's attached above, see https://github.com/user-attachments/files/20947873/full_log.txt. Or do you need something else?

I think the key difference is that "Clone Repository in Volume" uses WSL, while using the devcontainer CLI took place in Windows. And in WSL, docker buildx is used via [117673 ms] github.com/docker/buildx v0.14.0 171fcbeb69d67c90ba7f44f41a9e418f6a6ec1da. And there we run into proxy issues, but also into caching issues etc. I'd love just to run podman buildx when using the "Clone Repository in Volume" feature.

Thanks!