Bug: concurrency causes Premature destruction of containers and networks (e.g. w/pytest-xdist)

[skeletorXVI]

Describe the bug

Testcontainers behaves incredibly flaky when running tests in parallel with pytest-xdist. Specifically containers and networks are destroyed prematurely resulting in network connections being interrupted, ports being reassigned to different containers resulting in calls to the wrong application or instance.

To Reproduce

Note due to the flakiness you might need to tune to number of runs or repeat the test execution to see the errors.

Install the packages used for the tests.

pip install pytest==8.2 pytest-xdist==3.6.1 testcontainers==4.4 requests==2.31

Create a file named example.py

from rand import randrange
from time import sleep

import pytest
import requests
from testcontainers.core.network import Network
from testcontainers.postgres import PostgresContainer
from testcontainers.vault import VaultContainer


@pytest.mark.parametrize("run", range(30))
def test_healthcheck(run: int) -> None:
    with Network() as network:
        with PostgresContainer().with_network(network) as postgres_container:
            with VaultContainer().with_network(network).with_exposed_ports(8200) as vault_container:
                sleep(randrange(1, 3))  # Simulate work and give some time to prematurely stop to showcase buggy behavior
                response = requests.get(f"{vault_container.get_connection_url()}/v1/sys/health")
                assert response.status_code == 200

Run pytest with parallel test execution on a multicore CPU (at least 2 cores).

pytest -n auto example.py

Runtime environment

Linux 6.8.0-76060800daily20240311-generic
Python 3.10 3.11 3.12
CPU i7-1165G7
testcontainers 4.4

[alexanderankin]

What do you mean by Give some time to prematurely stop

…

On Thu, May 9, 2024, 2:57 PM Fabian Haenel ***@***.***> wrote: *Describe the bug* Testcontainers behaves incredibly flaky when running tests in parallel with pytest-xdist. Specifically containers and networks are destroyed prematurely resulting in network connections being interrupted, ports being reassigned to different containers resulting in calls to the wrong application or instance. *To Reproduce* Install the packages used for the tests. pip install pytest==8.2 pytest-xdist==3.6 testcontainers==4.4 requests==2.31 Create a file named example.py from time import sleep import pytestimport requestsfrom testcontainers.core.network import Networkfrom testcontainers.postgres import PostgresContainerfrom testcontainers.vault import VaultContainer @pytest.mark.parametrize("run", range(30))def test_healthcheck(run: int) -> None: with Network() as network: with PostgresContainer().with_network(network) as postgres_container: with VaultContainer().with_network(network).with_exposed_ports(8200) as vault_container: sleep(1) # Give some time to prematurely stop response = requests.get(f"{vault_container.get_connection_url()}/v1/sys/health") assert response.status_code == 200 Run pytest with parallel test execution on a multicore CPU (at least 2 cores). pytest -n auto example.py *Runtime environment* Linux 6.8.0-76060800daily20240311-generic Python 3.10 3.11 3.12 CPU i7-1165G7 testcontainers 4.4 — Reply to this email directly, view it on GitHub <#567>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACECGJEDDYQAU64UDVEMKRLZBPBJNAVCNFSM6AAAAABHPKJ4OOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI4DQMRXGI3DQNY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[skeletorXVI]

This is a minimal example to showcase the buggy behavior, in my actual tests the sleep is other operations that just take some time, in which case the container sometimes have been stopped and disposed already, when they should not have been (the context has not been left at that point).

Additional information:
I tested the same tests on a Raspberry Pi 4B and could not reproduce the errors.

[skeletorXVI]

Sometimes I see errors as follows, which are caused by the vault container in my example being stopped mid-test execution. It did not crash, it was stopped and removed before it should have been.

The issue is hard to reproduce, due to the issues flaky nature, sometimes no tests fail, sometimes only one, sometimes many.

==================================================================================================== test session starts =====================================================================================================
platform linux -- Python 3.10.12, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/fabian/BugReports
plugins: xdist-3.6.1
8 workers [30 items]    
..........................F...                                                                                                                                                                                         [100%]
========================================================================================================== FAILURES ==========================================================================================================
____________________________________________________________________________________________________ test_healthcheck[26] ____________________________________________________________________________________________________
[gw3] linux -- Python 3.10.12 /home/fabian/BugReports/venv-3.10/bin/python3.10

self = <urllib3.connectionpool.HTTPConnectionPool object at 0x745fd76001f0>, method = 'GET', url = '/v1/sys/health', body = None
headers = {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}, retries = Retry(total=0, connect=None, read=False, redirect=None, status=None)
redirect = False, assert_same_host = False, timeout = Timeout(connect=None, read=None, total=None), pool_timeout = None, release_conn = False, chunked = False, body_pos = None, preload_content = False
decode_content = False, response_kw = {}, parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/v1/sys/health', query=None, fragment=None), destination_scheme = None, conn = None, release_this_conn = True
http_tunnel_required = False, err = None, clean_exit = False

    def urlopen(  # type: ignore[override]
        self,
        method: str,
        url: str,
        body: _TYPE_BODY | None = None,
        headers: typing.Mapping[str, str] | None = None,
        retries: Retry | bool | int | None = None,
        redirect: bool = True,
        assert_same_host: bool = True,
        timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
        pool_timeout: int | None = None,
        release_conn: bool | None = None,
        chunked: bool = False,
        body_pos: _TYPE_BODY_POSITION | None = None,
        preload_content: bool = True,
        decode_content: bool = True,
        **response_kw: typing.Any,
    ) -> BaseHTTPResponse:
        """
        Get a connection from the pool and perform an HTTP request. This is the
        lowest level call for making a request, so you'll need to specify all
        the raw details.
    
        .. note::
    
           More commonly, it's appropriate to use a convenience method
           such as :meth:`request`.
    
        .. note::
    
           `release_conn` will only behave as expected if
           `preload_content=False` because we want to make
           `preload_content=False` the default behaviour someday soon without
           breaking backwards compatibility.
    
        :param method:
            HTTP request method (such as GET, POST, PUT, etc.)
    
        :param url:
            The URL to perform the request on.
    
        :param body:
            Data to send in the request body, either :class:`str`, :class:`bytes`,
            an iterable of :class:`str`/:class:`bytes`, or a file-like object.
    
        :param headers:
            Dictionary of custom headers to send, such as User-Agent,
            If-None-Match, etc. If None, pool headers are used. If provided,
            these headers completely replace any pool-specific headers.
    
        :param retries:
            Configure the number of retries to allow before raising a
            :class:`~urllib3.exceptions.MaxRetryError` exception.
    
            If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a
            :class:`~urllib3.util.retry.Retry` object for fine-grained control
            over different types of retries.
            Pass an integer number to retry connection errors that many times,
            but no other types of errors. Pass zero to never retry.
    
            If ``False``, then retries are disabled and any exception is raised
            immediately. Also, instead of raising a MaxRetryError on redirects,
            the redirect response will be returned.
    
        :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int.
    
        :param redirect:
            If True, automatically handle redirects (status codes 301, 302,
            303, 307, 308). Each redirect counts as a retry. Disabling retries
            will disable redirect, too.
    
        :param assert_same_host:
            If ``True``, will make sure that the host of the pool requests is
            consistent else will raise HostChangedError. When ``False``, you can
            use the pool on an HTTP proxy and request foreign hosts.
    
        :param timeout:
            If specified, overrides the default timeout for this one
            request. It may be a float (in seconds) or an instance of
            :class:`urllib3.util.Timeout`.
    
        :param pool_timeout:
            If set and the pool is set to block=True, then this method will
            block for ``pool_timeout`` seconds and raise EmptyPoolError if no
            connection is available within the time period.
    
        :param bool preload_content:
            If True, the response's body will be preloaded into memory.
    
        :param bool decode_content:
            If True, will attempt to decode the body based on the
            'content-encoding' header.
    
        :param release_conn:
            If False, then the urlopen call will not release the connection
            back into the pool once a response is received (but will release if
            you read the entire contents of the response such as when
            `preload_content=True`). This is useful if you're not preloading
            the response's content immediately. You will need to call
            ``r.release_conn()`` on the response ``r`` to return the connection
            back into the pool. If None, it takes the value of ``preload_content``
            which defaults to ``True``.
    
        :param bool chunked:
            If True, urllib3 will send the body using chunked transfer
            encoding. Otherwise, urllib3 will send the body using the standard
            content-length form. Defaults to False.
    
        :param int body_pos:
            Position to seek to in file-like body in the event of a retry or
            redirect. Typically this won't need to be set because urllib3 will
            auto-populate the value when needed.
        """
        parsed_url = parse_url(url)
        destination_scheme = parsed_url.scheme
    
        if headers is None:
            headers = self.headers
    
        if not isinstance(retries, Retry):
            retries = Retry.from_int(retries, redirect=redirect, default=self.retries)
    
        if release_conn is None:
            release_conn = preload_content
    
        # Check host
        if assert_same_host and not self.is_same_host(url):
            raise HostChangedError(self, url, retries)
    
        # Ensure that the URL we're connecting to is properly encoded
        if url.startswith("/"):
            url = to_str(_encode_target(url))
        else:
            url = to_str(parsed_url.url)
    
        conn = None
    
        # Track whether `conn` needs to be released before
        # returning/raising/recursing. Update this variable if necessary, and
        # leave `release_conn` constant throughout the function. That way, if
        # the function recurses, the original value of `release_conn` will be
        # passed down into the recursive call, and its value will be respected.
        #
        # See issue #651 [1] for details.
        #
        # [1] <https://github.com/urllib3/urllib3/issues/651>
        release_this_conn = release_conn
    
        http_tunnel_required = connection_requires_http_tunnel(
            self.proxy, self.proxy_config, destination_scheme
        )
    
        # Merge the proxy headers. Only done when not using HTTP CONNECT. We
        # have to copy the headers dict so we can safely change it without those
        # changes being reflected in anyone else's copy.
        if not http_tunnel_required:
            headers = headers.copy()  # type: ignore[attr-defined]
            headers.update(self.proxy_headers)  # type: ignore[union-attr]
    
        # Must keep the exception bound to a separate variable or else Python 3
        # complains about UnboundLocalError.
        err = None
    
        # Keep track of whether we cleanly exited the except block. This
        # ensures we do proper cleanup in finally.
        clean_exit = False
    
        # Rewind body position, if needed. Record current position
        # for future rewinds in the event of a redirect/retry.
        body_pos = set_file_position(body, body_pos)
    
        try:
            # Request a connection from the queue.
            timeout_obj = self._get_timeout(timeout)
            conn = self._get_conn(timeout=pool_timeout)
    
            conn.timeout = timeout_obj.connect_timeout  # type: ignore[assignment]
    
            # Is this a closed/new connection that requires CONNECT tunnelling?
            if self.proxy is not None and http_tunnel_required and conn.is_closed:
                try:
                    self._prepare_proxy(conn)
                except (BaseSSLError, OSError, SocketTimeout) as e:
                    self._raise_timeout(
                        err=e, url=self.proxy.url, timeout_value=conn.timeout
                    )
                    raise
    
            # If we're going to release the connection in ``finally:``, then
            # the response doesn't need to know about the connection. Otherwise
            # it will also try to release it and we'll have a double-release
            # mess.
            response_conn = conn if not release_conn else None
    
            # Make the request on the HTTPConnection object
>           response = self._make_request(
                conn,
                method,
                url,
                timeout=timeout_obj,
                body=body,
                headers=headers,
                chunked=chunked,
                retries=retries,
                response_conn=response_conn,
                preload_content=preload_content,
                decode_content=decode_content,
                **response_kw,
            )

venv-3.10/lib/python3.10/site-packages/urllib3/connectionpool.py:793: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv-3.10/lib/python3.10/site-packages/urllib3/connectionpool.py:537: in _make_request
    response = conn.getresponse()
venv-3.10/lib/python3.10/site-packages/urllib3/connection.py:466: in getresponse
    httplib_response = super().getresponse()
/usr/lib/python3.10/http/client.py:1375: in getresponse
    response.begin()
/usr/lib/python3.10/http/client.py:318: in begin
    version, status, reason = self._read_status()
/usr/lib/python3.10/http/client.py:279: in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <socket.SocketIO object at 0x745fd76b85b0>, b = <memory at 0x745fd7842140>

    def readinto(self, b):
        """Read up to len(b) bytes into the writable buffer *b* and return
        the number of bytes read.  If the socket is non-blocking and no bytes
        are available, None is returned.
    
        If *b* is non-empty, a 0 return value indicates that the connection
        was shutdown at the other end.
        """
        self._checkClosed()
        self._checkReadable()
        if self._timeout_occurred:
            raise OSError("cannot read from timed out object")
        while True:
            try:
>               return self._sock.recv_into(b)
E               ConnectionResetError: [Errno 104] Connection reset by peer

/usr/lib/python3.10/socket.py:705: ConnectionResetError

During handling of the above exception, another exception occurred:

self = <requests.adapters.HTTPAdapter object at 0x745fd76001c0>, request = <PreparedRequest [GET]>, stream = False, timeout = Timeout(connect=None, read=None, total=None), verify = True, cert = None
proxies = OrderedDict()

    def send(
        self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None
    ):
        """Sends PreparedRequest object. Returns Response object.
    
        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
        :param stream: (optional) Whether to stream the request content.
        :param timeout: (optional) How long to wait for the server to send
            data before giving up, as a float, or a :ref:`(connect timeout,
            read timeout) <timeouts>` tuple.
        :type timeout: float or tuple or urllib3 Timeout object
        :param verify: (optional) Either a boolean, in which case it controls whether
            we verify the server's TLS certificate, or a string, in which case it
            must be a path to a CA bundle to use
        :param cert: (optional) Any user-provided SSL certificate to be trusted.
        :param proxies: (optional) The proxies dictionary to apply to the request.
        :rtype: requests.Response
        """
    
        try:
            conn = self.get_connection(request.url, proxies)
        except LocationValueError as e:
            raise InvalidURL(e, request=request)
    
        self.cert_verify(conn, request.url, verify, cert)
        url = self.request_url(request, proxies)
        self.add_headers(
            request,
            stream=stream,
            timeout=timeout,
            verify=verify,
            cert=cert,
            proxies=proxies,
        )
    
        chunked = not (request.body is None or "Content-Length" in request.headers)
    
        if isinstance(timeout, tuple):
            try:
                connect, read = timeout
                timeout = TimeoutSauce(connect=connect, read=read)
            except ValueError:
                raise ValueError(
                    f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, "
                    f"or a single float to set both timeouts to the same value."
                )
        elif isinstance(timeout, TimeoutSauce):
            pass
        else:
            timeout = TimeoutSauce(connect=timeout, read=timeout)
    
        try:
>           resp = conn.urlopen(
                method=request.method,
                url=url,
                body=request.body,
                headers=request.headers,
                redirect=False,
                assert_same_host=False,
                preload_content=False,
                decode_content=False,
                retries=self.max_retries,
                timeout=timeout,
                chunked=chunked,
            )

venv-3.10/lib/python3.10/site-packages/requests/adapters.py:486: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv-3.10/lib/python3.10/site-packages/urllib3/connectionpool.py:847: in urlopen
    retries = retries.increment(
venv-3.10/lib/python3.10/site-packages/urllib3/util/retry.py:470: in increment
    raise reraise(type(error), error, _stacktrace)
venv-3.10/lib/python3.10/site-packages/urllib3/util/util.py:38: in reraise
    raise value.with_traceback(tb)
venv-3.10/lib/python3.10/site-packages/urllib3/connectionpool.py:793: in urlopen
    response = self._make_request(
venv-3.10/lib/python3.10/site-packages/urllib3/connectionpool.py:537: in _make_request
    response = conn.getresponse()
venv-3.10/lib/python3.10/site-packages/urllib3/connection.py:466: in getresponse
    httplib_response = super().getresponse()
/usr/lib/python3.10/http/client.py:1375: in getresponse
    response.begin()
/usr/lib/python3.10/http/client.py:318: in begin
    version, status, reason = self._read_status()
/usr/lib/python3.10/http/client.py:279: in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <socket.SocketIO object at 0x745fd76b85b0>, b = <memory at 0x745fd7842140>

    def readinto(self, b):
        """Read up to len(b) bytes into the writable buffer *b* and return
        the number of bytes read.  If the socket is non-blocking and no bytes
        are available, None is returned.
    
        If *b* is non-empty, a 0 return value indicates that the connection
        was shutdown at the other end.
        """
        self._checkClosed()
        self._checkReadable()
        if self._timeout_occurred:
            raise OSError("cannot read from timed out object")
        while True:
            try:
>               return self._sock.recv_into(b)
E               urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

/usr/lib/python3.10/socket.py:705: ProtocolError

During handling of the above exception, another exception occurred:

run = 26

    @pytest.mark.parametrize("run", range(30))
    def test_healthcheck(run: int) -> None:
        with Network() as network:
            with PostgresContainer().with_network(network) as postgres_container:
                with VaultContainer().with_network(network).with_exposed_ports(8200) as vault_container:
                    sleep(1)
>                   response = requests.get(f"{vault_container.get_connection_url()}/v1/sys/health")

example.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv-3.10/lib/python3.10/site-packages/requests/api.py:73: in get
    return request("get", url, params=params, **kwargs)
venv-3.10/lib/python3.10/site-packages/requests/api.py:59: in request
    return session.request(method=method, url=url, **kwargs)
venv-3.10/lib/python3.10/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
venv-3.10/lib/python3.10/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <requests.adapters.HTTPAdapter object at 0x745fd76001c0>, request = <PreparedRequest [GET]>, stream = False, timeout = Timeout(connect=None, read=None, total=None), verify = True, cert = None
proxies = OrderedDict()

    def send(
        self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None
    ):
        """Sends PreparedRequest object. Returns Response object.
    
        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
        :param stream: (optional) Whether to stream the request content.
        :param timeout: (optional) How long to wait for the server to send
            data before giving up, as a float, or a :ref:`(connect timeout,
            read timeout) <timeouts>` tuple.
        :type timeout: float or tuple or urllib3 Timeout object
        :param verify: (optional) Either a boolean, in which case it controls whether
            we verify the server's TLS certificate, or a string, in which case it
            must be a path to a CA bundle to use
        :param cert: (optional) Any user-provided SSL certificate to be trusted.
        :param proxies: (optional) The proxies dictionary to apply to the request.
        :rtype: requests.Response
        """
    
        try:
            conn = self.get_connection(request.url, proxies)
        except LocationValueError as e:
            raise InvalidURL(e, request=request)
    
        self.cert_verify(conn, request.url, verify, cert)
        url = self.request_url(request, proxies)
        self.add_headers(
            request,
            stream=stream,
            timeout=timeout,
            verify=verify,
            cert=cert,
            proxies=proxies,
        )
    
        chunked = not (request.body is None or "Content-Length" in request.headers)
    
        if isinstance(timeout, tuple):
            try:
                connect, read = timeout
                timeout = TimeoutSauce(connect=connect, read=read)
            except ValueError:
                raise ValueError(
                    f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, "
                    f"or a single float to set both timeouts to the same value."
                )
        elif isinstance(timeout, TimeoutSauce):
            pass
        else:
            timeout = TimeoutSauce(connect=timeout, read=timeout)
    
        try:
            resp = conn.urlopen(
                method=request.method,
                url=url,
                body=request.body,
                headers=request.headers,
                redirect=False,
                assert_same_host=False,
                preload_content=False,
                decode_content=False,
                retries=self.max_retries,
                timeout=timeout,
                chunked=chunked,
            )
    
        except (ProtocolError, OSError) as err:
>           raise ConnectionError(err, request=request)
E           requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

venv-3.10/lib/python3.10/site-packages/requests/adapters.py:501: ConnectionError
---------------------------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------------------------
Pulling image postgres:latest
Container started: 0d6d539ba38c
Waiting for container <Container: 0d6d539ba38c> with image postgres:latest to be ready ...
Pulling image hashicorp/vault:latest
Container started: 639c1b35b412
Waiting for container <Container: 639c1b35b412> with image hashicorp/vault:latest to be ready ...
Waiting for container <Container: 639c1b35b412> with image hashicorp/vault:latest to be ready ...
Waiting for container <Container: 639c1b35b412> with image hashicorp/vault:latest to be ready ...
----------------------------------------------------------------------------------------------------- Captured log call ------------------------------------------------------------------------------------------------------
INFO     testcontainers.core.container:container.py:88 Pulling image postgres:latest
INFO     testcontainers.core.container:container.py:100 Container started: 0d6d539ba38c
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 0d6d539ba38c> with image postgres:latest to be ready ...
INFO     testcontainers.core.container:container.py:88 Pulling image hashicorp/vault:latest
INFO     testcontainers.core.container:container.py:100 Container started: 639c1b35b412
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 639c1b35b412> with image hashicorp/vault:latest to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 639c1b35b412> with image hashicorp/vault:latest to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 639c1b35b412> with image hashicorp/vault:latest to be ready ...
================================================================================================== short test summary info ===================================================================================================
FAILED example.py::test_healthcheck[26] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
=============================================================================================== 1 failed, 29 passed in 56.28s ================================================================================================

[alexanderankin]

So I'm guessing something in the Ryuk class or something else is failing to start the containers under parallelism. I guess we could start surrounding parts of the code with threading.Locks to see if it helps?

[Tranquility2]

I'm going to go a bit off road here, @skeletorXVI are you familiar with pytest-xdist
--dist loadgroup flag? can be very helpful in some cases where you are running pytest -n and some library or even your own code is not fully ready or able to run in a distributed way.

Note

Taken from https://pytest-xdist.readthedocs.io/en/latest/distribution.html

--dist loadgroup: Tests are grouped by the xdist_group mark. Groups are distributed to available workers as whole units. This guarantees that all tests with same xdist_group name run in the same worker.

@alexanderankin I know this is not a proper fix, just trying to have some sort of workaround.

[alexanderankin]

@skeletorXVI in your recollection, is a 30 second delay relevant here? As in, could the containers be cleaned up after a 30 second delay and your tests take > 30 seconds to run? This would be explained by ryuk loosing connection to the application, which it then times out after a fixed delay (which iirc is 30 seconds).

[skeletorXVI]

@skeletorXVI in your recollection, is a 30 second delay relevant here? As in, could the containers be cleaned up after a 30 second delay and your tests take > 30 seconds to run? This would be explained by ryuk loosing connection to the application, which it then times out after a fixed delay (which iirc is 30 seconds).

The test suite in it's entirety take longer than 30 seconds but there are no individual tests that get even close to 30 seconds.

@Tranquility2
--dist loadgroup did not solve the issue either.

[jkp]

I'm seeing this very same problem with pytest-xdist and tests that use a postgres container each. For the sake of clarity: xdist puts each test in a separate subprocess so each test that uses a database requests its own container.

Increasing concurrency seems to exacerbate the problem...

[jkp]

Looking around it seems like folks have had the same issue in other languages. Assuming all languages use the same underlying container then there seems to be an issue with the code that waits for the container to start up...here is the fix they made for rust.

I'm going to monkeypatch our code and see if this resolves things locally.

[jkp]

OK - I can see how the python implementation would cause some indeterminate errors. The fix was to wait for the log message on both stderr AND stdout, one for each boot cycle - the problem with the python code is that it has an OR condition that only requires presence of the line in either log. Monkey patching this isn't the cleanest but I can knock something up to prove the fix - a proper fix would require a more involved patch to testcontainers and I'm not sure of the knock-on effects that might have.

[jkp]

I worked around the issue with the following patch in our pytest conftest.py:

"""
Monkey-patch for testcontainers-python/postgres issue
https://github.com/testcontainers/testcontainers-python/issues/567#issuecomment-2260676840
"""


def test_containers_wait_for_both_logs(
    container: postgres.PostgresContainer,
    predicate: Union[Callable, str],
    timeout: float = postgres.c.timeout,
    interval: float = 1,
) -> float:
    if isinstance(predicate, str):
        predicate = re.compile(predicate, re.MULTILINE).search
    start = time.time()
    while True:
        duration = time.time() - start
        stdout = container.get_logs()[0].decode()
        stderr = container.get_logs()[1].decode()
        if predicate(stdout) and predicate(stderr):
            return duration
        if duration > timeout:
            raise TimeoutError(f"Container did not emit logs satisfying predicate in {timeout:.3f} " "seconds")
        time.sleep(interval)


postgres.wait_for_logs = test_containers_wait_for_both_logs

[alexanderankin]

i can change it to an and rather than an or if you want, i observe it does get printed twice. i am trying to figure out how to observe whether it is stdout or stderr. but it does seem very plausible

[jkp]

The trouble is that method is used in other places in the codebase so its probably better to parametize it somehow...thats the reason I didn't attempt a more comprehensive patch. But you folks know this library better than me so whatever you think works - it fixes the issue anyway.

[skeletorXVI]

i can change it to an and rather than an or if you want, i observe it does get printed twice. i am trying to figure out how to observe whether it is stdout or stderr. but it does seem very plausible

The trouble is that method is used in other places in the codebase so its probably better to parametize it somehow...thats the reason I didn't attempt a more comprehensive patch. But you folks know this library better than me so whatever you think works - it fixes the issue anyway.

How about switching to a v2 with predicate(stdout, stderr), so that the determination of and and or is up to the implementation rather than relying on guesswork?

[alexanderankin]

oh i see the issue is with the wait_for_logs function not in the postgres module itself. yeah thats a bit tricky. i suspect that whole API will change with testcontainers v5 to look more like the oop ones (e.g. rust copies the "wait for" api).

[alexanderankin]

#661 will make it so you can do and with current api

[skeletorXVI]

From my testing the following PR solved this bug: #678