5.4 builds fail under Linux due to ZenServer timeout

[adamrehn]

When building Linux container images for Unreal Engine 5.4.1, I discovered that DDC generation fails due to a timeout when attempting to communicate with the local ZenServer instance:

ZenLocal: Readiness check failed. It will be deactivated until responsiveness improves. If this is consistent, consider disabling this cache store through the use of the '-ddc=NoZenLocalFallback' or '-ddc=InstalledNoZenLocalFallback' commandline arguments.

During the cook process, the Editor launches ZenServer as a child process and attempts to communicate with it via the IPv6 loopback address [::1]. Unfortunately, Docker does not enable IPv6 support by default, and users need to manually configure the Docker daemon to enable it. As a result, when building images under the default Docker daemon configuration, communication with the ZenServer fails and the cook process terminates unsuccessfully.

I've reached out to my contacts at Epic to see if this can be resolved for 5.5, but in the meantime, there are two ways to successfully build Linux images for Unreal Engine 5.4 with DDC enabled:

• Configure the Docker daemon to enable IPv6 for the default bridge network

• Enable host networking mode by passing --network=host to the docker build command, bypassing the bridge network and sidestepping the lack of IPv6 support

The latter option would be the easiest to implement in ue4-docker (and the most convenient for users), although the former is arguably the more "correct" solution. We could either add --network=host to our build commands under Linux, or modify the ue4-docker setup command to enable IPv6 support and require users to run that before building images for 5.4.

I'm currently leaning towards the --network=host option myself, do you have a preference @slonopotamus?

[slonopotamus]

Docker does not enable IPv6 support by default

There were multiple changes in recent docker versions wrt IPv6. See v26.0.0 release notes. I think it is enabled now.

The latter option would be the easiest to implement in ue4-docker (and the most convenient for users),

I personally do not use ue4-docker setup, --network=host is definitely easier for ue4-docker users, so I also vote for the latter option.

However I don't quite understand how --network=host is supposed to help if the host machine doesn't have IPv6.

[adamrehn]

There were multiple changes in recent docker versions wrt IPv6. See v26.0.0 release notes. I think it is enabled now.

Oh, nice! I guess that would be a third alternative then: we could specify 26.0.0 as the minimum required Docker version for building images for 5.4. That would actually be much nicer than either of my proposed options above, and we could simply document the need for the daemon configuration steps for those users who are unable to update to the latest version of Docker for one reason or another.

However I don't quite understand how --network=host is supposed to help if the host machine doesn't have IPv6.

I haven't tested in an IPv4-only environment to explicitly confirm this, but my understanding is that none of these options (configuring the daemon prior to 26.0.0, using host networking mode, or using 26.0.0) will allow communication over [::1] to work if the host kernel doesn't support IPv6. In that case, I think building Linux images for 5.4 with DDC enabled would be impossible, regardless of what we implement (unless we patch the ZenServer code in ue4-docker, but I'd really like to avoid patching things on our side, since that's always messy and causes potential issues when users are building images for their own forks of the engine).

[slonopotamus]

I've just tested docker 26.1.0 (my distro doesn't offer me 26.0.0 out of the box, and I just wanted to do a quick test). Indeed, there is an IPv6 address on loopback interface inside container without any additional efforts.

[slonopotamus]

After hitting exactly the same issue, I vote for requiring Docker >=26. Because users will need this anyway if they want to run their images, even if we workaround the build process with --network=host.

[TBBle]

Yeah, I was wondering about that (but wasn't sure, so didn't comment at the time), but since this will also affect using the container image outside the ue4-docker-controlled environment, setting a minimum Docker version seems less fragile than maintaining ZenServer source patches.

If they need it, someone can do a custom-source build from 5.4 with such patches if they want, so the version-requirement should not be a hard-blockage.

[adamrehn]

Okay, I've added a warning to the ue4-docker build command in commit 1aa50f2.

Given that this issue impacts users' ability to run cooks inside built images, do you think it would be prudent to include the warning in the ue4-docker info command as well, so users see it if they go to post an issue about runtime cook failures? It wouldn't necessarily help the maintainers in triaging new issues (we'll be able to spot the old Docker daemon version in the existing output anyway), but perhaps it might help to minimise the creation of duplicate issues?

[slonopotamus]

We can even add ue4-docker diagnostics!

[adamrehn]

Excellent idea! That way, users who are stuck using older versions of the Docker daemon and need to perform manual configuration changes to enable IPv6 can verify that the config changes are working as expected.

It's worth noting that the ping command doesn't seem to recognise the IPv6 loopback address ::1 (at least from what I saw in my testing when first diagnosing this issue), but a curl http://[::1] and a simple HTTP server should work as a test to verify that IPv6 support is functioning correctly.

[slonopotamus]

It's worth noting that the ping command doesn't seem to recognise the IPv6 loopback address ::1

You want to do ping6 ::1.

UPD: Hmm... For me, ping ::1 works perfectly inside ubuntu-22.04 container and docker 26.1.0...

[adamrehn]

Oh nice, I didn't know about ping6!

Weird, I just ran a fresh Ubuntu 22.04 container with Docker 26.1.4 and installed the inetutils-ping package, and the non-suffixed version doesn't resolve the loopback:

root@ac68c9b58680:/# ping ::1
ping: unknown host

root@ac68c9b58680:/# ping6 ::1
PING ::1 (::1): 56 data bytes
64 bytes from localhost: icmp_seq=0 ttl=64 time=0.053 ms
64 bytes from localhost: icmp_seq=1 ttl=64 time=0.072 ms
64 bytes from localhost: icmp_seq=2 ttl=64 time=0.099 ms

[adamrehn]

Okay, I've added an IPv6 diagnostic and updated the warning message to include instructions on how to run the diagnostic. I've also surfaced the warning in the ue4-docker info command in addition to the ue4-docker build command.

[slonopotamus]

So, looks like the issue can be closed.

Weird, I just ran a fresh Ubuntu 22.04 container with Docker 26.1.4 and installed the inetutils-ping package, and the non-suffixed version doesn't resolve the loopback

No idea what's going on here, exactly the same steps work for me. I am not that good at Linux networking to explain what affects this. I can mention that my box has a working IPv6 host network.