You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Client:
Context: default
Debug Mode: false
Server:
Containers: 3
Running: 1
Paused: 0
Stopped: 2
Images: 4
Server Version: 20.10.22
Storage Driver: fuse-overlayfs
Logging Driver: json-file
Cgroup Driver: none
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 78f51771157abb6c9ed224c22013cdf09962315d
runc version: v1.1.4-0-g5fd4c4d1
init version: de40ad0
Security Options:
seccomp
Profile: default
rootless
Kernel Version: 5.4.226-129.415.amzn2.x86_64
Operating System: Ubuntu 20.04.5 LTS (containerized)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.29GiB
Name: XXXXXXXXXXXXXXXXXX
ID: JP6C:YJBL:JFY5:YDGK:HAWT:Z6KA:IKSL:JKXP:4ZVG:JUB7:3TSG:GJHL
Docker Root Dir: /home/runner/.local/share/docker
Debug Mode: false
HTTP Proxy: XXXXXXXXXXXXXXXXXXXXX
HTTPS Proxy: XXXXXXXXXXXXXXXXXXXXXXXX
No Proxy: XXXXXXXXXXXXXXXXXXXX
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
What happened?
Starting point
I'm trying to get testcontainers working with Quarkus and docker-rootless on my self-hosted Github Action runner.
The GitHub Action runner is deployed via a k8s pod running on Kubernetes 1.22 & [Docker Engine ](docker://20.10.17). Within that container I've also installed docker-rootless (via the offical documentation), so no sidecar is being used (see Docker version for more details). Furthermore I'm using Quarkus to trigger my application tests which uses testcontainers to spin up required infrastructure services.
Problem description
The problem is that I'm unable to solve connection problems happening for containers managed by testcontainers. As you can see in the logs, the Quarkus test framework can't even communicate with Ryuk:
(testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
The same is true for any other service managed by testcontainers like redis (please see relevant log output).
My guess is it has something to do with my docker-rootless setup but I've no clue what could be the reason. When I start containers manually (e.g. via docker run ..) everything works as expected. So basically the docker-rootless installation works. I've also switched the same setup to an on-prem runner which is not running on Kubernetes but on a VM and uses docker-rootless as well. This setup works fine even when testcontainers comes into play, but obviously it's less complex (less layers of abstraction).
I've also figured out that it's by intention to communicate with the gateway instead of the containers ip directly. So I've tested whetherRyuk can ping 172.17.0.1. That' works. I've also spawned some new containers (busybox) within the same default network bridge and pinged Ryuk, also this works.
I've also tried to set TESTCONTAINERS_HOST_OVERRIDE=localhost as suggested in #4596. Without luck ... Anyway, in my case the containers are running on a container runtime within a single container( withn in a pod). IMHO the fact that containers share the same network within apod is not relevant for my nested setup, since Kubernetes (or in that case the outer docker runtime) is not aware of my nested containers.
Long story short: I'm a little bit frustrated and I don't know how to continue with my investigation. I just want to make sure it's not a testcontainer related issue. Please let me know if you need further logs or something else.
Any help would be very appreciated.
Relevant log output
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running XXXXXXTest
2023-01-30 15:57:17,366 WARN [org.tes.uti.TestcontainersConfiguration] (build-37) Attempted to read Testcontainers configuration file at file:/home/runner/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /home/runner/.testcontainers.properties (No such file or directory)
2023-01-30 15:57:17,843 INFO [org.tes.doc.DockerClientProviderStrategy] (build-37) Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///home/runner/.docker/run/docker.sock
2023-01-30 15:57:17,870 INFO [org.tes.uti.ImageNameSubstitutor] (build-37) Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
2023-01-30 15:57:17,898 INFO [doc.3.4]] (build-37) Pulling docker image: testcontainers/ryuk:0.3.4. Please be patient; this may take some time but only needs to be done once.
2023-01-30 15:57:19,463 INFO [doc.3.4]] (docker-java-stream-257369379) Starting to pull image
2023-01-30 15:57:19,490 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 0 pending, 0 downloaded, 0 extracted, (0 bytes/0 bytes)
2023-01-30 15:57:20,140 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 2 pending, 1 downloaded, 0 extracted, (317 KB/? MB)
2023-01-30 15:57:20,154 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 1 pending, 2 downloaded, 0 extracted, (317 KB/? MB)
2023-01-30 15:57:20,157 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 0 pending, 3 downloaded, 0 extracted, (317 KB/5 MB)
2023-01-30 15:57:20,303 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 0 pending, 3 downloaded, 1 extracted, (2 MB/5 MB)
2023-01-30 15:57:20,373 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 0 pending, 3 downloaded, 2 extracted, (2 MB/5 MB)
2023-01-30 15:57:20,487 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 0 pending, 3 downloaded, 3 extracted, (5 MB/5 MB)
2023-01-30 15:57:20,505 INFO [doc.3.4]] (docker-java-stream-257369379) Pull complete. 3 layers, pulled in 1s (downloaded 5 MB at 5 MB/s)
2023-01-30 15:57:20,538 INFO [doc.3.4]] (build-37) Creating container for image: testcontainers/ryuk:0.3.4
2023-01-30 15:57:20,614 INFO [doc.3.4]] (build-37) Container testcontainers/ryuk:0.3.4 is starting: 6d7021822af197cf173d859f70e77b5ea8de34121787b04035c04e162166816e
2023-01-30 15:57:20,942 INFO [doc.3.4]] (build-37) Container testcontainers/ryuk:0.3.4 started in PT3.057842824S
2023-01-30 15:57:20,954 INFO [doc.16]] (build-37) Pulling docker image: alpine:3.16. Please be patient; this may take some time but only needs to be done once.
2023-01-30 15:57:22,191 INFO [doc.16]] (docker-java-stream-1037873389) Starting to pull image
2023-01-30 15:57:22,192 INFO [doc.16]] (docker-java-stream-1037873389) Pulling image layers: 0 pending, 0 downloaded, 0 extracted, (0 bytes/0 bytes)
2023-01-30 15:57:22,885 INFO [doc.16]] (docker-java-stream-1037873389) Pulling image layers: 0 pending, 1 downloaded, 0 extracted, (28 KB/2 MB)
2023-01-30 15:57:22,991 INFO [doc.16]] (docker-java-stream-1037873389) Pulling image layers: 0 pending, 1 downloaded, 1 extracted, (2 MB/2 MB)
2023-01-30 15:57:28,403 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:33,663 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:38,916 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:44,172 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:49,427 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:53,406 ERROR [org.tes.uti.RyukResourceReaper] (build-37) Timed out waiting for Ryuk container to start. Ryuk's logs:2023/01/30 15:57:20 Pinging Docker...2023/01/30 15:57:20 Docker daemon is available!2023/01/30 15:57:20 Starting on port 8080...2023/01/30 15:57:20 Started!2023-01-30 15:57:53,417 INFO [doc.io/redis:7-alpine]] (build-37) Pulling docker image: docker.io/redis:7-alpine. Please be patient; this may take some time but only needs to be done once.2023-01-30 15:57:54,647 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Starting to pull image2023-01-30 15:57:54,648 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 0 pending, 0 downloaded, 0 extracted, (0 bytes/0 bytes)2023-01-30 15:57:54,682 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:57:55,322 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 5 pending, 1 downloaded, 0 extracted, (38 KB/? MB)2023-01-30 15:57:55,324 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 4 pending, 2 downloaded, 0 extracted, (38 KB/? MB)2023-01-30 15:57:55,344 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 3 pending, 3 downloaded, 0 extracted, (38 KB/? MB)2023-01-30 15:57:55,469 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 3 pending, 3 downloaded, 1 extracted, (3 MB/? MB)2023-01-30 15:57:55,503 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 3 pending, 3 downloaded, 2 extracted, (3 MB/? MB)2023-01-30 15:57:55,606 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 3 pending, 3 downloaded, 3 extracted, (3 MB/? MB)2023-01-30 15:57:55,665 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 2 pending, 4 downloaded, 3 extracted, (3 MB/? MB)2023-01-30 15:57:55,671 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 1 pending, 5 downloaded, 3 extracted, (3 MB/? MB)2023-01-30 15:57:55,831 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 0 pending, 6 downloaded, 3 extracted, (9 MB/11 MB)2023-01-30 15:57:56,089 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 0 pending, 6 downloaded, 4 extracted, (11 MB/11 MB)2023-01-30 15:57:56,121 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 0 pending, 6 downloaded, 5 extracted, (11 MB/11 MB)2023-01-30 15:57:56,160 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 0 pending, 6 downloaded, 6 extracted, (11 MB/11 MB)2023-01-30 15:57:56,177 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pull complete. 6 layers, pulled in 1s (downloaded 11 MB at 11 MB/s)2023-01-30 15:57:56,179 INFO [doc.io/redis:7-alpine]] (build-37) Creating container for image: docker.io/redis:7-alpine2023-01-30 15:57:56,207 INFO [doc.io/redis:7-alpine]] (build-37) Container docker.io/redis:7-alpine is starting: e94b722a880e478b2b07f2e03ac95151438a69cae75970bbd52614de1f03fa182023-01-30 15:57:59,936 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:05,191 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:10,446 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:15,701 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:20,955 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:26,210 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:31,465 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:36,720 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:41,975 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:47,227 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:52,482 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)2023-01-30 15:58:56,481 ERROR [doc.io/redis:7-alpine]] (build-37) Could not start container: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (172.17.0.1 ports: [32769] should be listening) at org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:102) at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52) at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:953) at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:485) at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344) at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81) at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334) at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322) at io.quarkus.redis.client.deployment.DevServicesRedisProcessor.lambda$startContainer$1(DevServicesRedisProcessor.java:181) at java.base/java.util.Optional.orElseGet(Optional.java:364) at io.quarkus.redis.client.deployment.DevServicesRedisProcessor.startContainer(DevServicesRedisProcessor.java:193) at io.quarkus.redis.client.deployment.DevServicesRedisProcessor.startRedisContainers(DevServicesRedisProcessor.java:103) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:909) at io.quarkus.builder.BuildContext.run(BuildContext.java:281) at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18) at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478) at java.base/java.lang.Thread.run(Thread.java:833) at org.jboss.threads.JBossThread.run(JBossThread.java:501)2023-01-30 15:58:56,505 ERROR [doc.io/redis:7-alpine]] (build-37) Log output from the failed container:1:C 30 Jan 2023 15:57:56.528 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo1:C 30 Jan 2023 15:57:56.528 # Redis version=7.0.8, bits=64, commit=00000000, modified=0, pid=1, just started1:C 30 Jan 2023 15:57:56.528 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf1:M 30 Jan 2023 15:57:56.528 * monotonic clock: POSIX clock_gettime1:M 30 Jan 2023 15:57:56.529 * Running mode=standalone, port=6379.1:M 30 Jan 2023 15:57:56.529 # Server initialized1:M 30 Jan 2023 15:57:56.529 * Ready to accept connections
I haven't figured out why the gateway ip is sometimes missing at the default network config, but testcontainers determines the ip properly as shown in the logs .
Debug output of dockerd-rootless:
+ [ -w /home/runner/.docker/run ]
+ [ -d /home/runner ]
+ rootlesskit=
+ command -v docker-rootlesskit
+ command -v rootlesskit
+ rootlesskit=rootlesskit
+ break
+ [ -z rootlesskit ]
+ :
+ :
+ : builtin
+ : auto
+ : auto
+ net=
+ mtu=
+ [ -z ]
+ command -v slirp4netns
+ slirp4netns --help
+ grep -qw -- --netns-type
+ net=slirp4netns
+ [ -z ]
+ mtu=65520
+ [ -z slirp4netns ]
+ [ -z 65520 ]
+ [ -z ]
+ _DOCKERD_ROOTLESS_CHILD=1
+ export _DOCKERD_ROOTLESS_CHILD
+ id -u
+ [ 1000 = 0 ]
+ command -v selinuxenabled
+ exec rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /home/runner/bin/dockerd-rootless.sh
WARN[0000] The host root filesystem is mounted as "master:158". Setting child propagation to "rslave" is not supported.
+ [ -w /home/runner/.docker/run ]
+ [ -d /home/runner ]
+ rootlesskit=
+ command -v docker-rootlesskit
+ command -v rootlesskit
+ rootlesskit=rootlesskit
+ break
+ [ -z rootlesskit ]
+ :
+ :
+ : builtin
+ : auto
+ : auto
+ net=
+ mtu=
+ [ -z ]
+ command -v slirp4netns
+ slirp4netns --help
+ grep -qw -- --netns-type
+ net=slirp4netns
+ [ -z ]
+ mtu=65520
+ [ -z slirp4netns ]
+ [ -z 65520 ]
+ [ -z 1 ]
+ [ 1 = 1 ]
+ rm -f /run/docker /run/containerd /run/xtables.lock
+ [ -n ]
+ stat -c %T -f /etc
+ [ tmpfs = tmpfs ]
+ [ -L /etc/ssl ]
+ realpath /etc/ssl
+ realpath_etc_ssl=/etc/.ro3276320264/ssl
+ rm -f /etc/ssl
+ mkdir /etc/ssl
+ mount --rbind /etc/.ro3276320264/ssl /etc/ssl
+ exec dockerd
INFO[2023-01-30T16:26:19.427007894Z] Starting up
WARN[2023-01-30T16:26:19.427052714Z] Running in rootless mode. This mode has feature limitations.
INFO[2023-01-30T16:26:19.427059194Z] Running with RootlessKit integration
INFO[2023-01-30T16:26:19.428038072Z] libcontainerd: started new containerd process pid=2067
INFO[2023-01-30T16:26:19.428073583Z] parsed scheme: "unix" module=grpc
INFO[2023-01-30T16:26:19.428082773Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2023-01-30T16:26:19.428108174Z] ccResolverWrapper: sending update to cc: {[{unix:///home/runner/.docker/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2023-01-30T16:26:19.428122444Z] ClientConn switching balancer to "pick_first" module=grpc
WARN[0000] containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header
INFO[2023-01-30T16:26:19.441135860Z] starting containerd revision=78f51771157abb6c9ed224c22013cdf09962315d version=v1.6.13
INFO[2023-01-30T16:26:19.449417591Z] loading plugin "io.containerd.content.v1.content"... type=io.containerd.content.v1
INFO[2023-01-30T16:26:19.449485393Z] loading plugin "io.containerd.snapshotter.v1.aufs"... type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.449619075Z] skip loading plugin "io.containerd.snapshotter.v1.aufs"... error="aufs is not supported (modprobe aufs failed: exec: \"modprobe\": executable file not found in $PATH \"\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.449642695Z] loading plugin "io.containerd.snapshotter.v1.btrfs"... type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.449876760Z] skip loading plugin "io.containerd.snapshotter.v1.btrfs"... error="path /home/runner/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (overlay) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.
containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.449897440Z] loading plugin "io.containerd.snapshotter.v1.devmapper"... type=io.containerd.snapshotter.v1
WARN[2023-01-30T16:26:19.449916451Z] failed to load plugin io.containerd.snapshotter.v1.devmapper error="devmapper not configured"
INFO[2023-01-30T16:26:19.449930241Z] loading plugin "io.containerd.snapshotter.v1.native"... type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.449971081Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"... type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.450448240Z] loading plugin "io.containerd.snapshotter.v1.zfs"... type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.450659404Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"... error="path /home/runner/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapsho
tter.v1
INFO[2023-01-30T16:26:19.450680494Z] loading plugin "io.containerd.metadata.v1.bolt"... type=io.containerd.metadata.v1
WARN[2023-01-30T16:26:19.450706705Z] could not use snapshotter devmapper in metadata plugin error="devmapper not configured"
INFO[2023-01-30T16:26:19.450721295Z] metadata content store policy set policy=shared
INFO[2023-01-30T16:26:19.450814497Z] loading plugin "io.containerd.differ.v1.walking"... type=io.containerd.differ.v1
INFO[2023-01-30T16:26:19.450837027Z] loading plugin "io.containerd.event.v1.exchange"... type=io.containerd.event.v1
INFO[2023-01-30T16:26:19.450858148Z] loading plugin "io.containerd.gc.v1.scheduler"... type=io.containerd.gc.v1
INFO[2023-01-30T16:26:19.450894578Z] loading plugin "io.containerd.service.v1.introspection-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.450914639Z] loading plugin "io.containerd.service.v1.containers-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.450933729Z] loading plugin "io.containerd.service.v1.content-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.450951809Z] loading plugin "io.containerd.service.v1.diff-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.450972330Z] loading plugin "io.containerd.service.v1.images-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.451275365Z] loading plugin "io.containerd.service.v1.leases-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.451291665Z] loading plugin "io.containerd.service.v1.namespaces-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.451326006Z] loading plugin "io.containerd.service.v1.snapshots-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.451346797Z] loading plugin "io.containerd.runtime.v1.linux"... type=io.containerd.runtime.v1
INFO[2023-01-30T16:26:19.451415608Z] loading plugin "io.containerd.runtime.v2.task"... type=io.containerd.runtime.v2
INFO[2023-01-30T16:26:19.451509280Z] loading plugin "io.containerd.monitor.v1.cgroups"... type=io.containerd.monitor.v1
INFO[2023-01-30T16:26:19.451828415Z] loading plugin "io.containerd.service.v1.tasks-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.451856836Z] loading plugin "io.containerd.grpc.v1.introspection"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.451868376Z] loading plugin "io.containerd.internal.v1.restart"... type=io.containerd.internal.v1
INFO[2023-01-30T16:26:19.451924057Z] loading plugin "io.containerd.grpc.v1.containers"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.451944477Z] loading plugin "io.containerd.grpc.v1.content"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.451968248Z] loading plugin "io.containerd.grpc.v1.diff"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.451985218Z] loading plugin "io.containerd.grpc.v1.events"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.451998838Z] loading plugin "io.containerd.grpc.v1.healthcheck"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.452016569Z] loading plugin "io.containerd.grpc.v1.images"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.452038209Z] loading plugin "io.containerd.grpc.v1.leases"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.452054899Z] loading plugin "io.containerd.grpc.v1.namespaces"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.452075720Z] loading plugin "io.containerd.internal.v1.opt"... type=io.containerd.internal.v1
WARN[2023-01-30T16:26:19.453204180Z] failed to load plugin io.containerd.internal.v1.opt error="mkdir /opt/containerd: permission denied"
INFO[2023-01-30T16:26:19.453305902Z] loading plugin "io.containerd.grpc.v1.snapshots"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.453328433Z] loading plugin "io.containerd.grpc.v1.tasks"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.453339983Z] loading plugin "io.containerd.grpc.v1.version"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.453349313Z] loading plugin "io.containerd.tracing.processor.v1.otlp"... type=io.containerd.tracing.processor.v1
INFO[2023-01-30T16:26:19.453361433Z] skip loading plugin "io.containerd.tracing.processor.v1.otlp"... error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
INFO[2023-01-30T16:26:19.453371783Z] loading plugin "io.containerd.internal.v1.tracing"... type=io.containerd.internal.v1
ERRO[2023-01-30T16:26:19.453392684Z] failed to initialize a tracing processor "otlp" error="no OpenTelemetry endpoint: skip plugin"
INFO[2023-01-30T16:26:19.453554907Z] serving... address=/home/runner/.docker/run/docker/containerd/containerd-debug.sock
INFO[2023-01-30T16:26:19.453640908Z] serving... address=/home/runner/.docker/run/docker/containerd/containerd.sock.ttrpc
INFO[2023-01-30T16:26:19.453710789Z] serving... address=/home/runner/.docker/run/docker/containerd/containerd.sock
INFO[2023-01-30T16:26:19.453740410Z] containerd successfully booted in 0.013205s
INFO[2023-01-30T16:26:19.460450613Z] parsed scheme: "unix" module=grpc
INFO[2023-01-30T16:26:19.460473513Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2023-01-30T16:26:19.460500384Z] ccResolverWrapper: sending update to cc: {[{unix:///home/runner/.docker/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2023-01-30T16:26:19.460513474Z] ClientConn switching balancer to "pick_first" module=grpc
INFO[2023-01-30T16:26:19.461329809Z] parsed scheme: "unix" module=grpc
INFO[2023-01-30T16:26:19.461351739Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2023-01-30T16:26:19.461366499Z] ccResolverWrapper: sending update to cc: {[{unix:///home/runner/.docker/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2023-01-30T16:26:19.461377109Z] ClientConn switching balancer to "pick_first" module=grpc
INFO[2023-01-30T16:26:19.462047952Z] [graphdriver] using prior storage driver: fuse-overlayfs
WARN[2023-01-30T16:26:19.479733634Z] Your kernel does not support cgroup blkio weight
WARN[2023-01-30T16:26:19.479752304Z] Your kernel does not support cgroup blkio weight_device
INFO[2023-01-30T16:26:19.479885997Z] Loading containers: start.
INFO[2023-01-30T16:26:19.492071069Z] skipping firewalld management for rootless mode
INFO[2023-01-30T16:26:19.546072302Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2023-01-30T16:26:19.576222692Z] Loading containers: done.
INFO[2023-01-30T16:26:19.579787597Z] Docker daemon commit=42c8b31 graphdriver(s)=fuse-overlayfs version=20.10.22
INFO[2023-01-30T16:26:19.579846848Z] Daemon has completed initialization
INFO[2023-01-30T16:26:19.595244968Z] API listen on /home/runner/.docker/run/docker.sock
The text was updated successfully, but these errors were encountered:
I have to admit that TESTCONTAINERS_HOST_OVERRIDE=localhost actually works. At least for my functional testcontainers, but not for Ryuk. However, the latter one is disabled anyway since I'm using ephemeral pods for GutHub Action runners only. Thus this issue can be closed.
Module
Core
Testcontainers version
The one being bundeled with Quarkus 2.15.3.Final
Using the latest Testcontainers version?
Yes
Host OS
Ubuntu 20.04.5 LTS (containerized)
Host Arch
x86_64
Docker version
What happened?
Starting point
I'm trying to get
testcontainers
working withQuarkus
anddocker-rootless
on my self-hostedGithub Action runner
.The
GitHub Action runner
is deployed via ak8s pod
running onKubernetes 1.22 & [Docker Engine ](docker://20.10.17)
. Within that container I've also installeddocker-rootless
(via the offical documentation), so no sidecar is being used (see Docker version for more details). Furthermore I'm usingQuarkus
to trigger myapplication tests
which usestestcontainers
to spin up required infrastructure services.Problem description
The problem is that I'm unable to solve connection problems happening for containers managed by
testcontainers
. As you can see in the logs, theQuarkus
test framework can't even communicate withRyuk
:(testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
The same is true for any other service managed by
testcontainers
likeredis
(please see relevant log output).My guess is it has something to do with my
docker-rootless
setup but I've no clue what could be the reason. When I start containers manually (e.g. viadocker run ..
) everything works as expected. So basically thedocker-rootless
installation works. I've also switched the same setup to anon-prem runner
which is not running onKubernetes
but on aVM
and usesdocker-rootless
as well. This setup works fine even whentestcontainers
comes into play, but obviously it's less complex (less layers of abstraction).I've also figured out that it's by intention to communicate with the
gateway
instead of thecontainers ip
directly. So I've tested whetherRyuk
can ping172.17.0.1
. That' works. I've also spawned some new containers (busybox) within the same default networkbridge
and pingedRyuk
, also this works.I've also tried to set
TESTCONTAINERS_HOST_OVERRIDE=localhost
as suggested in #4596. Without luck ... Anyway, in my case the containers are running on acontainer runtime
within a single container( withn in a pod).IMHO
the fact that containers share the same network within apod
is not relevant for my nested setup, sinceKubernetes (or in that case the outer docker runtime)
is not aware of my nested containers.Long story short: I'm a little bit frustrated and I don't know how to continue with my investigation. I just want to make sure it's not a
testcontainer
related issue. Please let me know if you need further logs or something else.Any help would be very appreciated.
Relevant log output
Additional Information
I haven't figured out why the
gateway ip
is sometimes missing at the default network config, buttestcontainers
determines theip
properly as shown in the logs .Debug output of dockerd-rootless:
The text was updated successfully, but these errors were encountered: