test_interpreters: test_create_many_threaded() failed on FreeBSD: log: RuntimeError: interpreter creation failed #109700
Comments
|
Bug also seen on aarch64 RHEL8 LTO + PGO 3.x: https://buildbot.python.org/all/#/builders/78/builds/5402 |
|
Important part in output: |
It comes from the FreeBSD build. The aarch64 RHEL8 LTO + PGO 3.x build has a different error: |
|
I failed to reproduce the issue on Linux just with this command: |
This looks very suspicious. Why two |
That's how aarch64 RHEL8 LTO + PGO 3.x is configured.
Yes, there are two |
|
It may be related to #109615. What if run tests locally in such configuration? |
|
This test consumes a lot of memory. 200 threads need more than 6 GB of memory if run them simultaneous (and if not, then what is the point of using so many threads?). Perhaps there is a leak, because if run tests repeatedly with limited memory, they finally crash. 200 live interpreters also consume sufficient amount of memory. All normal tests only need 600-700 MB of memory. Tests which need more are marked with @ericsnowcurrently What is the purpose of this test? Can it use less threads, for example 10 threads, each sequentially creating 20 interpreters? Can 5 threads be enough? |
|
test_create_many_sequential also crashes due to leaks. If you double the limit, it will crash on the 7th iteration instead of the 2nd. If triple -- on 11th. So it leaks 1.5-2 MB per interpreter. These two tests ran sequentially can leak up to 6 GB of memory. |
|
Wow, so now it's possible to leak a whole interpreter? Maybe we need something like threading_setup() / threading_cleanup() which uses _thread._count(), to count how many interpreters we have before/after running tests? |
Actually, only up to 600 MB, but it is much anyway. I do not know how it is now, but several years ago some of buildbots had only few hundreds of MBs of physical memory, often failed after hours of swapping due to timeout. I suppose buildbots on which test_create_many_threaded fails also have very limited RAM. |
|
More precisely, 1.4 MB are leaked in every subinterpreter. |
Almost a floppy disk (1.44 MB)! |
|
Sorry I didn't see this sooner. I don't know why, but the GitHub notifications page isn't showing me this issue. Thankfully @serhiy-storchaka DM'ed me.
The purpose of the test is to make sure we don't crash if we create a bunch of subinterpreters and keep them alive at the same time. We try it both one-at-a-time and in parallel (though the GIL keeps it mostly one-at-a-time for now). The number of interpreters (hence the number of threads) needs to be high enough that we can consistently trigger possible races in interpreter initialization and other resource contention. FWIW, the stress tests were inspired by some code @tonybaloney shared with me earlier this year and I was using to hunt down crashes: (expand)import time
import _xxsubinterpreters as _interpreters
from threading import Thread
from queue import Queue
import os
def run(host: str, port: int, results: Queue):
# Create a communication channel
r, w = os.pipe()
interpid = _interpreters.create()
subinterpreters.run_string(interpid, f"""if True:
import os
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(1)
result = sock.connect_ex(({host!r}, {port}))
os.write({w}, result.to_bytes())
sock.close()
""")
print("completed", flush=True)
output = os.read(r, 10)
if int.from_bytes(output) == 0:
results.put(port)
if __name__ == '__main__':
start = time.time()
host = "localhost" # pick a friend
threads = []
results = Queue()
for port in range(1, 100):
# for port in range(80, 100):
t = Thread(target=run, args=(host, port, results))
t.start()
threads.append(t)
# t.join()
for t in threads:
t.join()
while not results.empty():
print("Port {0} is open".format(results.get()))
print("Completed scan in {0} seconds".format(time.time() - start))When I started, I got semi-reliable crashes if I bumped the number of threads from 20 to 100. So that's what I did in the test. I don't recall why I bumped it up to 200 for the threaded version. IIRC, I toyed with scaling that down if the host had limited RAM, but decided against the extra complexity. Maybe I should revisit that. Do we need that many interpreters/threads in the stress tests? Maybe not. I just want to be sure we give ourselves the best chance to find crashes. While we figure that out, at the least we could apply the
That would definitely be worth looking into.
AFAICS, each interpreter uses a bit over 2 MB. Not all of that is necessarily resident. When an interpreter is finalized, the system allocator determines how much of that process memory to actually release to the system. That makes it harder to determine what leaks there may be. That said, it certainly seems like something's leaking. |
|
@ericsnowcurrently: What's the status of this issue? Should it be closed? |
|
I still need to investigate this further. It's certainly still a problem. Sorry it's taking so long. |
ericsnowcurrently
added
type-bug
|
I didn't see this error recently so I close the issue. |
* Ensure that destructors are called in the test that created interpreters, not after finishing it. * Try to create/run interpreters in threads simultaneously. * Mark tests that requires over 6GB of memory with bigmemtest.
…nGH-109946) * Ensure that destructors are called in the test that created interpreters, not after finishing it. * Try to create/run interpreters in threads simultaneously. * Mark tests that requires over 6GB of memory with bigmemtest. (cherry picked from commit 61b50a9) Co-authored-by: Serhiy Storchaka <[email protected]>
…09946) (GH-133391) * Ensure that destructors are called in the test that created interpreters, not after finishing it. * Try to create/run interpreters in threads simultaneously. * Mark tests that requires over 6GB of memory with bigmemtest. (cherry picked from commit 61b50a9) Co-authored-by: Serhiy Storchaka <[email protected]>
* origin/main: (111 commits) pythongh-91048: Add filename and line number to external inspection routines (pythonGH-133385) pythongh-131178: Add tests for `ast` command-line interface (python#133329) Regenerate pcbuild.sln in Visual Studio 2022 (python#133394) pythongh-133042: disable HACL* HMAC on Emscripten (python#133064) pythongh-133351: Fix remote PDB's multi-line block tab completion (python#133387) pythongh-109700: Improve stress tests for interpreter creation (pythonGH-109946) pythongh-81793: Skip tests for os.link() to symlink on Android (pythonGH-133388) pythongh-126835: Rename `ast_opt.c` to `ast_preprocess.c` and related stuff after moving const folding to the peephole optimizier (python#131830) pythongh-91048: Relax test_async_global_awaited_by to fix flakyness (python#133368) pythongh-132457: make staticmethod and classmethod generic (python#132460) pythongh-132805: annotationlib: Fix handling of non-constant values in FORWARDREF (python#132812) pythongh-132426: Add get_annotate_from_class_namespace replacing get_annotate_function (python#132490) pythongh-81793: Always call linkat() from os.link(), if available (pythonGH-132517) pythongh-122559: Synchronize C and Python implementation of the io module about pickling (pythonGH-122628) pythongh-69605: Add PyREPL import autocomplete feature to 'What's New' (python#133358) bpo-44172: Keep reference to original window in curses subwindow objects (pythonGH-26226) pythonGH-133231: Changes to executor management to support proposed `sys._jit` module (pythonGH-133287) pythongh-133363: Fix Cmd completion for lines beginning with `! ` (python#133364) pythongh-132983: Introduce `_zstd` bindings module (pythonGH-133027) pythonGH-91048: Add utils for printing the call stack for asyncio tasks (python#133284) ...
|
There were many |
|
But the leak in interpreters has not been fixed. $ (ulimit -v 300000; ./python -u -c '
from test.support import interpreters
for i in range(1000):
print(i)
interpreters.create()
')
...
250
251
RuntimeError: Failed to import encodings module
During handling of the above exception, another exception occurred:
interpreters.InterpreterError: sub-interpreter creation failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 5, in <module>
interpreters.create()
~~~~~~~~~~~~~~~~~~~^^
File "/home/serhiy/py/cpython/Lib/test/support/interpreters/__init__.py", line 76, in create
id = _interpreters.create(reqrefs=True)
interpreters.InterpreterError: interpreter creation failed
python: Objects/typeobject.c:297: managed_static_type_state_clear: Assertion `!_PyRuntime.types.managed_static.types[full_index].interp_count' failed. |
build: https://buildbot.python.org/all/#/builders/1223/builds/187
Linked PRs
The text was updated successfully, but these errors were encountered: