Add application-scope setting to control passing --use-host-proxy

[cortex93]

• VSCode Version: 1.51.0
• Local OS Version: windows 10 1809
• Remote OS Version: Linux 30278701a030 5.4.39-linuxkit #1 SMP Fri May 8 23:03:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Remote Extension/Connection Type: Docker

Steps to Reproduce:

1. Clean the local %TMP%/vsch folder to force download of the server package
2. Start a container (cmd): docker run --name vscode-proxy-test -d ubuntu sleep infinity
3. Set a vscode-server machine settings.json in the container
   (cmd): docker exec vscode-proxy-test /bin/sh -c "mkdir -p /root/.vscode-server/data/Machine && echo '{\"http.proxy\": \"http://no.whre:3128\"}' > /root/.vscode-server/data/Machine/settings.json"
4. Attach the container with vscode : PASSED
5. Dettach, clean the local %TMP%/vsch folder, (cmd): docker exec -it vscode-proxy-test -c "rm -rf /root/.vscode-server/bin/"
6. Attache the container with vscode : FAILED

[1709 ms] Start: Run in container: test -d /root/.vscode-server/bin/fcac248b077b55bae4ba5bab613fd6e9156c2f0c
[1711 ms] 
[1711 ms] 
[1711 ms] Exit code 1
[1711 ms] Installing VS Code Server for commit fcac248b077b55bae4ba5bab613fd6e9156c2f0c
[1711 ms] Start: Run in container: mkdir -p /root/.vscode-server/bin/fcac248b077b55bae4ba5bab613fd6e9156c2f0c_1605087259898
[1714 ms] 
[1714 ms] 
[1714 ms] Start: Downloading VS Code Server
[1719 ms] Error: getaddrinfo ENOTFOUND no.whre
        at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:60:26)

I expect the download to be done either localy without proxy settings set in the container or in the container with the proxy settings.

[chrmarti]

These settings are not read at the time the server is downloaded. Did you add the proxy also to .vscode/settings.json in the workspace folder?

[cortex93]

Proxy is defined in Remote settings (~/.vscode-server/data/Machine/settings.json inside container) and in the named container configuration file on host.
When attaching to the container, i'm not in a workspace.

I'm trying to reproduce with insiders but I can't figure out how to force the download again.

[fsackur]

@cortex93 if you delete or rename the relevant folder in .vscode-server/bin, it will have to re-download. The current release commit is fcac248b077b55bae4ba5bab613fd6e9156c2f0c - you can get this from About > Help, or by running code -v

[cortex93]

@cortex93 if you delete or rename the relevant folder in .vscode-server/bin, it will have to re-download. The current release commit is fcac248b077b55bae4ba5bab613fd6e9156c2f0c - you can get this from About > Help, or by running code -v

@fsackur Deleting the whole .vscode-server in the container do not redownload the package.

[7475 ms] Start: Run in container: test -d /home/vscode/.vscode-server/bin/fcac248b077b55bae4ba5bab613fd6e9156c2f0c
[7479 ms] 
[7479 ms] 
[7479 ms] Exit code 1
[7480 ms] Installing VS Code Server for commit fcac248b077b55bae4ba5bab613fd6e9156c2f0c
[7480 ms] Start: Run in container: mkdir -p /home/vscode/.vscode-server/bin/fcac248b077b55bae4ba5bab613fd6e9156c2f0c_1605021801978
[7487 ms] 
[7488 ms] 
[7561 ms] Start: Run in container: (dd iflag=fullblock bs=8192 count=3687 2>/dev/null; dd iflag=fullblock bs=5516 count=1 2>/dev/null) | tar --no-same-owner -xz --strip-components 1 -C /home/vscode/.vscode-server/bin/fcac248b077b55bae4ba5bab613fd6e9156c2f0c_1605021801978

[chrmarti]

Maybe you set the proxy with the local HTTP_PROXY environment variable or the user settings? Machine and folder settings are not read at the time the server is downloaded.

The last log output you posted suggests it installed the server and did not download again. Did it work?

[cortex93]

I have updated the steps to reproduce

[chrmarti]

@cortex93 The machine settings you add are not read at the time the server is downloaded. They become available only later. You must have some other local configuration that triggers this.

[chrmarti]

@sandy081 Are you remembering the machine and folder settings from previously opening a remote URI? So they are available while the resolver is still running?

[sandy081]

Yes, I cache remote machine and folder settings and use them when opening the remote URI again.

[cortex93]

@cortex93 The machine settings you add are not read at the time the server is downloaded. They become available only later. You must have some other local configuration that triggers this.

I updated steps to reproduce which should now be accurate. The configuration is effectively cached somewhere (based on the container name?) and reused locally where it should not.

[cortex93]

Any updates on this ?
I saw that machine settings are now printed right before server package download.
It seems also that the archive is not cached anymore on the host part. Every container, I open needs to download the package which means I need to modify the proxy settings for each before attaching, and set the value back when updated.

[chrmarti]

Downloads should still be cached, could you retry and confirm?

Nothing has changed here yet. A possible fix would be to ignore the machine setting on the local extension host when connected to a remote (currently that layers on top of the user setting).

[chrmarti]

Filed microsoft/vscode#119466 to track.

[cortex93]

Downloads should still be cached, could you retry and confirm?

Nothing has changed here yet. A possible fix would be to ignore the machine setting on the local extension host when connected to a remote (currently that layers on top of the user setting).

File is cached but seems to be deleted after a while. So if you don't update all your container at once, latter it will fail again.

[cortex93]

It seems it has been fixed in 1.55.0. Didn't have to edit my host file.

[fsackur]

1.55.0 also fixes the issue for me and at least one colleague. Shall we close this?

[chrmarti]

Maybe a side-effect of bug #4812.

[cortex93]

Weirdly enough,, updating to 1.55.1 didn't work again for me.

[fsackur]

We also find 1.55.1 regresses this issue. :-(

[cortex93]

Hi, things are getting worst since a few versions. The proxy settings seems now to be only global. When in remote container, I must configure a proxy the container can connect so that extension can work from inside the container. But then, everything vscode does from the host is not working (like browsing extension marketplace, checking for updates).

To add some context to this issue: a lot of companies are using proxy with NTLM authentication but a lot of tools like vscode or majority of linux software do not handle such an authent mecanism. One solution is to run locally a cntlm proxy container and use it as a local proxy. The issue arise because as it run under docker for windows, there is no common hostname (that I know of) to used which resolve to an ip the container is listening to. And thus, there is a difference between the uri used inside other docker containers and on the host.

[chrmarti]

@sandy081 It looks like while the resolver is running the http.proxy setting is read from the cache of the machine settings. In the local extension host, I would expect this setting to always be read from the user settings.

[felwolff]

This really seems to be the case. If I change my hosts file on Windows so it maps host.docker.internal to 127.0.0.1, downloading the server succeeds.
However, there seems to be more issues afterwards. When I install the Microsoft C/C++ extension, it downloads some additional stuff after installation. But that download seems to use the Proxy configured in the User settings, i.e. it tries to connect to 127.0.0.1:3128 from within the container. Changing the Proxy user settings to host.docker.internal:3128 fixes that, but that shouldn't be the User Proxy setting (not sure if that's an issue of the extension or VS Code itself though).

[chrmarti]

A local dev container is using the user setting, not the machine setting. That makes sense when the proxy is another machine in your network and has the advantage of not requiring any extra configuration in the dev container.

[felwolff]

What do you mean with machine setting? If I configure a proxy in the User settings, I configure the proxy IP that is reachable from Windows, i.e. 127.0.0.1. However, this IP is not reachable from within the local dev container, because there 127.0.0.1 does not reach the Windows host but the loopback of the container itself. Therefore the proxy used inside the dev container should not be the same as the proxy configured in the User settings that is supposed to work on Windows. Or am I missing something here?

[chrmarti]

Machine setting is the remote setting (the Remote tab in the settings editor).

I agree, it doesn't work for the localhost address or only with the workaround you found.

[cortex93]

The problem is that proxy settings is now only a user settings (proxy setting doesn't show up in the remote tab, neither workspace) and if you manually set it in the machine/settings.json inside the container you got a linter warning This setting can be applied only in application user settings. (Same for workspace settings.json).

Which mean the same value is used on the host and inside the container. And this is a big issue, because as stated since the begining of this issue, due to the way networking works, when the proxy is actually running on the host as a docker container, it can only be reached from the host on loopback address and in the container on the host.docker.internal (which resolve differently in the container and on the host).

It seems, one version of vscode around 1.55.0 was actually working as one would expect, which is either :

• using the proxy settings from host and run the resolve and download from the host (which would only work for downloading the package but will block extension in the container)
• using the proxy settings of the remote container (from machine/settings.json) and run the resolve and download from within the container. (which would work for extension running in the container)

@sandy081 It looks like while the resolver is running the http.proxy setting is read from the cache of the machine settings. In the local extension host, I would expect this setting to always be read from the user settings.

Yes, you are right. Fixing this will make things more consistent but it will not help to resolve the issue. At least, we should be able to download the update package by settings the proxy setting to a host reachable value. It will fail starting extension that need to reach the proxy. So we would need to change again the proxy settings to a container reachable value and restart the remote session. And voilà.

There is definitely something wrong in assuming the proxy settings has the same value everywhere. And since this design decision, proxy setting must be changed depending you want the host being able to make request or the container extensions. I'm using container or wsl locally only but remote ssl to a remote machine will certainly have different proxy settings.

Microsoft vscode is currently not able to handle Microsoft NTLM protocol. One solution would be to support NTLM proxy authentification inside vscode which is why we are using a CNTLM proxy on the host in the first place. If vscode and extension can handle the host proxy configuration, it wouldn't be an issue. But I guess is it will be easier to correctly manage different proxy settings on the host and in remote container, because it seems a lot of extensions currently relies on linux binaries that are clearly not NTLM friendly.
Another solution would be to have vscode install a proxy reachable from the remote container and chains the request to the host which then use the user proxy setting.

[cortex93]

Some good news.

I open a remote ssh connection (vscode 1.59.0 on both host and remote). And proxy settings can be edited in the remote tab (not in workspace but it's fair to me). No linter warning when manually editing the file.

So I set a non existent proxy in the remote tab like http://localhost/, delete the vscode-server bin directory, and reconnect.

It download the vscode server from the host like it should (not using the remote proxy setting).
I then browse the extension marketplace, still not using the remote proxy setting, and install humao.rest-client.
I tried a GET https://bing.com from the remote connection. And boom, as expected, I got Failed to establish a socket connection to proxies: ["PROXY localhost"] which shows extension are using proxy settings of the remote machine settings. What a relief.

So, all seems to be working as expected with remote SSL.

The question, now, is why Remote WSL/Container work differently ?

[felwolff]

Until the proxy stuff works as expected inside the container when using a local authentication proxy on the host, I have found a workaround that seems to be quite stable for me. Maybe it helps someone else as well.
Inside my container, I install and run tinyproxy. It listens inside the container on 127.0.0.1:3218 and forwards the traffic to host.docker.internal:3128. Therefore I can live with the User settings proxy pointing to http://127.0.0.1:3128, because now there is a proxy running on that address both inside the container and on the host.

There were some issues however starting the proxy in the container on time, so that the extensions can be installed while the container is being setup. Therefore I did the following:

Dockerfile

FROM ubuntu:18.04

RUN apt-get update \
    && apt-get install -y \
        tinyproxy

# create the tinyproxy conf in you .devcontainer directory
COPY tinyproxy.conf /etc/tinyproxy/tinyproxy.conf
# copy self-signed certificates to /usr/local/share/ca-certificates/ if necessary
RUN update-ca-certificates

# start the tinyproxy in the forground at the end of the dockerfile
CMD ["tinyproxy", "-d"]

The important parts of the tinyproxy conf are:

• Listen 127.0.0.1
• Port 3128
• Upstream host.docker.internal:3128

devcontainer.json

{
    "name": "DockerDevEnv",
    "build": {
        "dockerfile": "Dockerfile",
        "args": {
            // while building our container, it needs access to the proxy running on the host in order to install packages
            "http_proxy": "http://host.docker.internal:3128",
            "https_proxy": "http://host.docker.internal:3128"
        }
    },
    "runArgs": [
        "--cap-add=SYS_PTRACE",
        "--security-opt",
        "seccomp=unconfined"
    ],
    // Set *default* container specific settings.json values on container create.
    "settings": {},
    "containerEnv": {
        // once the container is running, it connects to our tinyproxy locally
        "http_proxy": "http://127.0.0.1:3128",
        "https_proxy": "http://127.0.0.1:3128",
        // only necessary if you installed self-signed certificates
        "NODE_EXTRA_CA_CERTS": "/etc/ssl/certs/ca-certificates.crt"
    },
    "extensions": [
        // add extension as usual
    ],
    // in order to run our CMD from the Dockerfile
    "overrideCommand": false
}

[cortex93]

@felwolff you may omit the build args and containerEnv proxy settings by configuring the docker client proxy (https://docs.docker.com/network/proxy/). This will also make your builded images and devcontainer.json more portable like usable on an env not requiring a proxy.

Your solution is fine in a closed team with same constraint. But when sharing a public repo it doesn't work. For eg, all examples provided by Microsoft at https://github.com/Microsoft/vscode-dev-containers just do not work.

[scrocquesel]

Some good news.

I open a remote ssh connection (vscode 1.59.0 on both host and remote). And proxy settings can be edited in the remote tab (not in workspace but it's fair to me). No linter warning when manually editing the file.

So I set a non existent proxy in the remote tab like http://localhost/, delete the vscode-server bin directory, and reconnect.

It download the vscode server from the host like it should (not using the remote proxy setting).
I then browse the extension marketplace, still not using the remote proxy setting, and install humao.rest-client.
I tried a GET https://bing.com from the remote connection. And boom, as expected, I got Failed to establish a socket connection to proxies: ["PROXY localhost"] which shows extension are using proxy settings of the remote machine settings. What a relief.

So, all seems to be working as expected with remote SSL.

The question, now, is why Remote WSL/Container work differently ?

@chrmarti Can you confirm with the source code, that the behavior is different between remote ssl and remote wsl/container ?

[sandy081]

@chrmarti There is a long thread to follow. May I know if this gets fixed if microsoft/vscode#119466 is supported?

[cortex93]

@chrmarti There is a long thread to follow. May I know if this gets fixed if microsoft/vscode#119466 is supported?

Things should be the same as for Remote SSL, as it seems the behavior is not the same.

Since this issue was opened, the proxy settings is no more overridable in Remote WSL/Container context so if we set the proxy setting at the host level, updates works as expected but then it leads to another issue because extensions running inside the container will not work if the proxy is not reachable from the container (for eg extension humao.rest-client).

Remote SSL allows to override proxy setting inside the remote ssh session (machine/settings) which allow for extension to use a different proxy than the host. Download occurs on the host side with the host proxy settings. So everything is working.

I think that for WSL and Container part, as they usually live on the same physical machine, it was assumed that the host and the remote part should share the same proxy settings which is not true.

[chrmarti]

@sandy081 I think you're right, microsoft/vscode#119466 will fix part of it, let's continue tracking that there. (What's confusing is that while the resolver is running the setting is always machine scoped. Only when the server starts do you learn whether we pass --use-host-proxy or not.)

Remote-Containers passes --use-host-proxy when we think that the container is running locally. That's assuming that a local container can use the same network configuration as the host. We should add an application-scope setting for Remote-Containers to control this. Tracking that here.

[sclorng]

Hi,

Have you any estimate about when this will at last be fixed ?

[sclorng]

@chrmarti I'm stuck with the last update 1.64.1.

[4396 ms] 
[4397 ms] Start: Run in container: cat /etc/passwd
[4403 ms] Start: Run in container: test -d /home/vscode/.vscode-server
[4406 ms] 
[4406 ms] 
[4407 ms] Start: Run in container: test ! -f '/home/vscode/.vscode-server/data/Machine/.writeMachineSettingsMarker' && set -o noclobber && mkdir -p '/home/vscode/.vscode-server/data/Machine' && { > '/home/vscode/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[4412 ms] 
[4413 ms] 
[4413 ms] Exit code 1
[4414 ms] Start: Run in container: cat /home/vscode/.vscode-server/data/Machine/settings.json
[4420 ms] {

 "http.proxy": "http://host.docker.internal:3128/"
}
[4421 ms] 
[4422 ms] Start: Run in container: test -d /home/vscode/.vscode-server/bin/d6ee99e4c045a6716e5c653d7da8e9ae6f5a8b03
[4429 ms] 
[4430 ms] 
[4431 ms] Exit code 1
[4433 ms] Installing VS Code Server for commit d6ee99e4c045a6716e5c653d7da8e9ae6f5a8b03
[4434 ms] Start: Downloading VS Code Server
[4435 ms] d6ee99e4c045a6716e5c653d7da8e9ae6f5a8b03 linux-x64 stable 
[25470 ms] Error: connect ETIMEDOUT 51.144.164.215:443
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16)
[25470 ms] Exit code ETIMEDOUT

A proxy is configured on the user settings, in the machine/settings.json and in the named config. None of them is used and I can't find a way to configure it.

It there a way to manually download the file ?

[sclorng]

Thanks to better logs in Remote WSL, I found out how to download it manually at http://update.code.visualstudio.com/commit:<sha>/server-linux-x64/stable.
Unpack the tgz to /home/vscode/.vscode-server/bin/d6ee99e4c045a6716e5c653d7da8e9ae6f5a8b03 and reload the container.

[chrmarti]

The issue in 1.64.1 is tracked as microsoft/vscode#142201. A fix will be in the upcoming VS Code 1.64.2.

[vscodenpa]

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.

If you wonder what we are up to, please see our roadmap and issue reporting guidelines.

Thanks for your understanding, and happy coding!