Authentication failed for 'https://github.com/EpicGames/UnrealEngine.git/'

[wohlbier]

Output of the ue4-docker info command:

[jgwohlbier@etc-gpu-09 ue4-docker]$ ue4-docker info
ue4-docker version:         0.0.102 (latest available version is 0.0.102)
Operating system:           Linux (Red Hat Enterprise Linux 8.7 (Ootpa), 4.18.0-425.3.1.el8.x86_64)
Docker daemon version:      20.10.21
NVIDIA Docker supported:    Yes
Maximum image size:         No limit detected
Available disk space:       3.99 TiB
Total system memory:        187.28 GiB physical, 0 bytes virtual
CPU:                        20 physical, 40 logical (x86_64)

Additional details:

• Are you accessing the network through a proxy server? Yes

I'm using a token that I have verified allows me to clone https://github.com/EpicGames/UnrealEngine.git.

[jgwohlbier@etc-gpu-09 ue4-docker]$ ue4-docker build 4.27.0 --cuda=11.4.0 
[ue4-docker build] COMMAND-LINE INVOCATION:
[ue4-docker build] ['ue4-docker', '4.27.0', '--cuda=11.4.0']

[ue4-docker build] UNREAL ENGINE VERSION SETTINGS:
[ue4-docker build] Custom build:  No
[ue4-docker build] Release:       4.27.0
[ue4-docker build] Repository:    https://github.com/EpicGames/UnrealEngine.git
[ue4-docker build] Branch/tag:    4.27.0-release

[ue4-docker build] ADVANCED CONFIGURATION OPTIONS:
[ue4-docker build] excluded_components: {"ddc": false, "debug": false, "templates": false}

[ue4-docker build] LINUX CONTAINER SETTINGS
[ue4-docker build] Base OS image: nvidia/cudagl:11.4.0-devel-ubuntu18.04

[ue4-docker build] GENERAL SETTINGS
[ue4-docker build] Build targets: build-prerequisites full minimal source
[ue4-docker build] Changelist override: 17155196
[ue4-docker build] Not excluding any Engine components.

Retrieving the Git credentials that will be used to clone the UE4 repo
Username: wohlbier
Password: 


[ue4-docker build] Image "adamrehn/ue4-build-prerequisites:cudagl11.4.0-ubuntu18.04" exists and rebuild not requested, skipping build.

[ue4-docker build] Building image "adamrehn/ue4-source:4.27.0-cudagl11.4.0-ubuntu18.04"...
[+] Building 1.5s (8/17)                                                        
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 4.50kB                                     0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 87B                                           0.0s
 => [internal] load metadata for docker.io/adamrehn/ue4-build-prerequisit  0.0s
 => [ 1/13] FROM docker.io/adamrehn/ue4-build-prerequisites:cudagl11.4.0-  0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 377B                                          0.0s
 => CACHED [ 2/13] COPY --chown=ue4:ue4 git-credential-helper-endpoint.sh  0.0s
 => [ 3/13] RUN chmod +x /tmp/git-credential-helper-endpoint.sh && echo '  0.4s
 => ERROR [ 4/13] RUN mkdir /home/ue4/UnrealEngine &&  cd /home/ue4/Unrea  1.0s
------                                                                          
 > [ 4/13] RUN mkdir /home/ue4/UnrealEngine && 	cd /home/ue4/UnrealEngine && 	git init && 	git remote add origin "https://github.com/EpicGames/UnrealEngine.git" && 	git fetch --progress --depth 1 origin "4.27.0-release" && 	git checkout FETCH_HEAD && echo '' && echo 'RUN directive complete. Docker will now commit the filesystem layer to disk.' && echo 'Note that for large filesystem layers this can take quite some time.' && echo 'Performing filesystem layer commit...' && echo '':
#8 0.422 Initialized empty Git repository in /home/ue4/UnrealEngine/.git/
#8 0.956 remote: Support for password authentication was removed on August 13, 2021.
#8 0.956 remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
#8 0.956 fatal: Authentication failed for 'https://github.com/EpicGames/UnrealEngine.git/'
------
executor failed running [/bin/sh -c mkdir /home/ue4/UnrealEngine && 	cd /home/ue4/UnrealEngine && 	git init && 	git remote add origin "$GIT_REPO" && 	git fetch --progress --depth 1 origin "$GIT_BRANCH" && 	git checkout FETCH_HEAD && echo '' && echo 'RUN directive complete. Docker will now commit the filesystem layer to disk.' && echo 'Note that for large filesystem layers this can take quite some time.' && echo 'Performing filesystem layer commit...' && echo '']: exit code: 128
[ue4-docker build] Error: failed to build image "adamrehn/ue4-source:4.27.0-cudagl11.4.0-ubuntu18.04".

[wohlbier]

I see that there is a token field in CredentialEndpoint.py, but there doesn't appear to be a way to send it in on the command line.

[slonopotamus]

Do not enter your GitHub password. Instead, enter GitHub access token with repo scope.

[wohlbier]

I am entering a token into the password field. I have confirmed that the token works to clone the repository.

…

On Sat, Dec 10, 2022, 11:14 AM Marat Radchenko ***@***.***> wrote: Do not enter your GitHub password. Instead, enter GitHub access token with repo scope. — Reply to this email directly, view it on GitHub <#290 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDU3JTYCBEDNDFIMLXEVODWMTB65ANCNFSM6AAAAAAS2J5RYQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[slonopotamus]

Okay, let's check one more thing. Are you using classic token (obtainable at https://github.com/settings/tokens/new)? You only need to add repo scope for it and nothing else. Generated token has a form of ghp_<random letters> string.

[wohlbier]

Yes, I think so. I selected only "repo" and that selects the ones below it. There is nothing else below it selected.

[slonopotamus]

I'm out of ideas then. The only case when GitHub says "Support for password authentication was removed on August 13, 2021" I am aware of is when user tries to use password for authentication.

[wohlbier]

I'm definitely not using my password. I'll try to find the place in the code where the authentication request is happening and repeat the exact authentication steps manually to try to get some clues.

…

On Mon, Dec 12, 2022, 6:18 AM Marat Radchenko ***@***.***> wrote: I'm out of ideas then. The only case when GitHub says "Support for password authentication was removed on August 13, 2021" I am aware of is when user tries to use *password* for authentication. — Reply to this email directly, view it on GitHub <#290 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDU3JTP65NVSMFJXAOJH7DWM4Q2NANCNFSM6AAAAAAS2J5RYQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[wohlbier]

I have determined that the build is getting to

ue4-docker/ue4docker/dockerfiles/ue4-source/linux/Dockerfile

Line 70 in 15cf89d

RUN mkdir /home/ue4/UnrealEngine && \

.
I am not sure if this is expected behavior, or it should have entered the other if branch {% if credential_mode == "secrets" %}. I haven't yet been able to determine where the credential_mode is set or how to see the key values pairs in git_config.items(). Does this help at all?

[wohlbier]

Moreover, the git_config if statements are not activating, if indeed that is where the user/token is supposed to happen.

ue4-docker/ue4docker/dockerfiles/ue4-source/linux/Dockerfile

Lines 70 to 80 in 15cf89d

	RUN mkdir /home/ue4/UnrealEngine && \
	cd /home/ue4/UnrealEngine && \
	git init && \
	{% if git_config %}
	{% for key, value in git_config.items() %}
	git config {{ key }} {{ value }} && \
	{% endfor %}
	{% endif %}
	git remote add origin "$GIT_REPO" && \
	git fetch --progress --depth 1 origin "$GIT_BRANCH" && \
	git checkout FETCH_HEAD

[wohlbier]

Is there some assumption built in about how credentials are being managed? I couldn't get it to go with the code that is there. With the following change to the Dockerfile I got it to go. It's easy to see how this approach could be made to work with command line argument substitutions.

--- a/ue4docker/dockerfiles/ue4-source/linux/Dockerfile
+++ b/ue4docker/dockerfiles/ue4-source/linux/Dockerfile
@@ -75,7 +75,7 @@ RUN mkdir /home/ue4/UnrealEngine && \
        git config {{ key }} {{ value }} && \
        {% endfor %}
        {% endif %}
-       git remote add origin "$GIT_REPO" && \
+       git remote add origin "https://wohlbier:<TOKEN>@github.com/EpicGames/UnrealEngine.git" && \
        git fetch --progress --depth 1 origin "$GIT_BRANCH" && \
        git checkout FETCH_HEAD 

[TBBle]

It's not the git_config, it's the preceeding block that sets up git inside the container to query back to the ue4-docker process for the username/password.

ue4-docker/ue4docker/dockerfiles/ue4-source/linux/Dockerfile

Lines 64 to 67 in 15cf89d

	# Install our git credential helper that forwards requests to the credential HTTP endpoint on the host
	COPY --chown=ue4:ue4 git-credential-helper-endpoint.sh /tmp/git-credential-helper-endpoint.sh
	ENV GIT_ASKPASS=/tmp/git-credential-helper-endpoint.sh
	RUN chmod +x /tmp/git-credential-helper-endpoint.sh

git_config here is for cases where you need to add other settings to the git config before the checkout, e.g., disabling TLS certificate verification if passing through a badly-behaved https proxy.

If the GIT_ASKPASS block wasn't working, then I expect it'd be prompting for a username/password inside the container (and hanging since we don't have a TTY attached) but it might be possible to add some -v or similar to the git fetch call to ensure it is talking to the password service correctly.

[adamrehn]

@wohlbier the approach depicted in your diff will leak your GitHub credentials into the image history, which is a security risk that ue4-docker goes to great lengths to avoid. Credentials are passed to git by means of a credential helper script, with separate helper script implementations for the two different credential modes: credential endpoint mode and BuildKit secrets mode. As @TBBle mentioned above, the helper script is configured using the GIT_ASKPASS environment variable, so you'd see a failed attempt to prompt for credentials if that wasn't being triggered.

The first question that needs to be answered is which credential mode (and thus which helper script) is being used, since the subsequent steps for debugging any problems will vary based on the specific implementation. If you run ue4-docker build with the --verbose flag, it should print the docker build command that is being used to build the ue4-source image. The command will provide a clear indication of which credential mode is being used.

Here's an example of what the command looks like when using credential endpoint mode (note the presence of the HOST_ADDRESS_ARG and HOST_TOKEN_ARG build arguments):

[ue4-docker build] Building image "adamrehn/ue4-source:5.1.0-opengl-ubuntu18.04"...
['docker', 'build', '-t', 'adamrehn/ue4-source:5.1.0-opengl-ubuntu18.04', '-t', 'adamrehn/ue4-source:5.1.0', '/usr/local/lib/python3.10/dist-packages/ue4docker/dockerfiles/ue4-source/linux', '--platform', 'linux/amd64', '--build-arg', 'NAMESPACE=adamrehn', '--build-arg', 'PREREQS_TAG=opengl-ubuntu18.04', '--build-arg', 'GIT_REPO=https://github.com/EpicGames/UnrealEngine.git', '--build-arg', 'GIT_BRANCH=5.1.0-release', '--build-arg', 'VERBOSE_OUTPUT=1', '--build-arg', 'HOST_ADDRESS_ARG=10.115.11.203', '--build-arg', 'HOST_TOKEN_ARG=d118e94eb4281b10860746553b77c485', '--file', '/tmp/tmpsgmjk9vi/dockerfiles/ue4-source/linux/Dockerfile']

And here's an example of what the command looks like when using BuildKit secrets mode (note the presence of the two --secret flags):

[ue4-docker build] Building image "adamrehn/ue4-source:5.1.0-opengl-ubuntu18.04"...
['docker', 'build', '-t', 'adamrehn/ue4-source:5.1.0-opengl-ubuntu18.04', '-t', 'adamrehn/ue4-source:5.1.0', '/usr/local/lib/python3.10/dist-packages/ue4docker/dockerfiles/ue4-source/linux', '--progress=plain', '--platform', 'linux/amd64', '--build-arg', 'NAMESPACE=adamrehn', '--build-arg', 'PREREQS_TAG=opengl-ubuntu18.04', '--build-arg', 'GIT_REPO=https://github.com/EpicGames/UnrealEngine.git', '--build-arg', 'GIT_BRANCH=5.1.0-release', '--build-arg', 'VERBOSE_OUTPUT=1', '--secret', 'id=username,src=/tmp/tmppt01b_qb/username', '--secret', 'id=password,src=/tmp/tmppt01b_qb/password', '--file', '/tmp/tmpvsnxvox4/dockerfiles/ue4-source/linux/Dockerfile']

Could you please paste the build command that ue4-docker prints for the ue4-source image when running your build with the --verbose flag?

[wohlbier]

Here's the output from that command with --verbose. I'm not seeing the same verbose output that you are. It's using the endpoint not the secret. I'm using a tmp.py driving script so that I don't need to have the program installed.

[jgwohlbier@etc-gpu-09 ue4-docker]$ cat tmp.py 
import ue4docker

ue4docker.main()

[jgwohlbier@etc-gpu-09 ue4-docker]$ python3 tmp.py build 4.27.0 --cuda=11.4.0 --verbose
[tmp.py build] COMMAND-LINE INVOCATION:
[tmp.py build] ['tmp.py', '4.27.0', '--cuda=11.4.0', '--verbose']

[tmp.py build] UNREAL ENGINE VERSION SETTINGS:
[tmp.py build] Custom build:  No
[tmp.py build] Release:       4.27.0
[tmp.py build] Repository:    https://github.com/EpicGames/UnrealEngine.git
[tmp.py build] Branch/tag:    4.27.0-release

[tmp.py build] ADVANCED CONFIGURATION OPTIONS:
[tmp.py build] excluded_components: {"ddc": false, "debug": false, "templates": false}

[tmp.py build] LINUX CONTAINER SETTINGS
[tmp.py build] Base OS image: nvidia/cudagl:11.4.0-devel-ubuntu18.04

[tmp.py build] GENERAL SETTINGS
[tmp.py build] Build targets: build-prerequisites full minimal source
[tmp.py build] Changelist override: 17155196
[tmp.py build] Not excluding any Engine components.

Retrieving the Git credentials that will be used to clone the UE4 repo
Username: wohlbier
Password: 


[tmp.py build] Image "adamrehn/ue4-build-prerequisites:cudagl11.4.0-ubuntu18.04" exists and rebuild not requested, skipping build.

[tmp.py build] Building image "adamrehn/ue4-source:4.27.0-cudagl11.4.0-ubuntu18.04"...
[+] Building 1.6s (8/17)                                                        
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 4.50kB                                     0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 93B                                           0.0s
 => [internal] load metadata for docker.io/adamrehn/ue4-build-prerequisit  0.0s
 => [ 1/13] FROM docker.io/adamrehn/ue4-build-prerequisites:cudagl11.4.0-  0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 401B                                          0.0s
 => CACHED [ 2/13] COPY --chown=ue4:ue4 git-credential-helper-endpoint.sh  0.0s
 => [ 3/13] RUN chmod +x /tmp/git-credential-helper-endpoint.sh && echo '  0.4s
 => ERROR [ 4/13] RUN mkdir /home/ue4/UnrealEngine &&  cd /home/ue4/Unrea  1.1s
------                                                                          
 > [ 4/13] RUN mkdir /home/ue4/UnrealEngine &&  cd /home/ue4/UnrealEngine &&   git init &&      git remote add origin "https://github.com/EpicGames/UnrealEngine.git" &&        git fetch --progress --depth 1 origin "4.27.0-release" &&      git checkout FETCH_HEAD && echo '' && echo 'RUN directive complete. Docker will now commit the filesystem layer to disk.' && echo 'Note that for large filesystem layers this can take quite some time.' && echo 'Performing filesystem layer commit...' && echo '':
#8 0.380 Initialized empty Git repository in /home/ue4/UnrealEngine/.git/
#8 1.049 remote: Support for password authentication was removed on August 13, 2021.
#8 1.049 remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
#8 1.049 fatal: Authentication failed for 'https://github.com/EpicGames/UnrealEngine.git/'
------
executor failed running [/bin/sh -c mkdir /home/ue4/UnrealEngine && 	cd /home/ue4/UnrealEngine && 	git init && 	git remote add origin "$GIT_REPO" && 	git fetch --progress --depth 1 origin "$GIT_BRANCH" && 	git checkout FETCH_HEAD && echo '' && echo 'RUN directive complete. Docker will now commit the filesystem layer to disk.' && echo 'Note that for large filesystem layers this can take quite some time.' && echo 'Performing filesystem layer commit...' && echo '']: exit code: 128
[tmp.py build] Error: failed to build image "adamrehn/ue4-source:4.27.0-cudagl11.4.0-ubuntu18.04".

[adamrehn]

@wohlbier okay, that narrows things down a bit. What happens if you specify --opt credential_mode=secrets when running the build command?

[wohlbier]

@adamrehn that appears to be working ... I swore I had tried that already.

[adamrehn]

@wohlbier okay, that suggests there's something about your system's network configuration that's interfering with the credential endpoint then (e.g. firewall rules, proxy settings, etc.) If you run the ue4-docker setup command, does it mention anything about configuring the firewall?

@slonopotamus @TBBle given that we always use BuildKit when building Linux containers these days, should we simply set secrets mode as the default credential mode for Linux containers? It provides better caching behaviour for rebuilds due to the absence of an ever-changing token, and it'd alleviate a number of firewall concerns that we currently have to worry about by default.

[TBBle]

Yeah, I think secrets mode should be the default, I hadn't realised we hadn't already made that change when making BuildKit required, since it's one of the big wins from BuildKit support.

[slonopotamus]

I think secrets mode should be the default

+1 from me.

[slonopotamus]

Additional thoughts: we've had BuildKit enabled on Linux for several months already and didn't get any negative feedback (well, except log truncation, but we can fix that via ue4-docker setup?). So it sounds very sane to continue that route.

[adamrehn]

well, except log truncation, but we can fix that via ue4-docker setup?

We could definitely configure this automatically, although it'll probably need to vary based on how different distros package the Docker daemon. Under Ubuntu, I edit /lib/systemd/system/docker.service to add the relevant environment variables to the systemd service definition:

[Service]
Environment="BUILDKIT_STEP_LOG_MAX_SIZE=1073741824"
Environment="BUILDKIT_STEP_LOG_MAX_SPEED=1073741824"

After editing the file, I restart the Docker Daemon by running the following systemd-specific commands:

systemctl daemon-reload
systemctl restart docker.service