test_frame does crash randomly on Linux (x86-64, ppc64le, s390x) #133261

vstinner · 2025-05-01T15:53:58Z

Crash report

Example: https://buildbot.python.org/#/builders/64/builds/9301

test_repr_deep (test.test_frame.FrameLocalsProxyMappingTests.test_repr_deep) ...

Fatal Python error: Segmentation fault

Current thread 0x00007f2ed21d5400 [python] (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/case.py", line 247 in handle
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/case.py", line 813 in assertRaises
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/mapping_tests.py", line 634 in test_repr_deep
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/case.py", line 615 in _callTestMethod
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/case.py", line 669 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/case.py", line 725 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/suite.py", line 122 in run
  ...

Linked PRs

The text was updated successfully, but these errors were encountered:

vstinner · 2025-05-01T16:10:20Z

The crash might be related to the recent trashcan change: 44e4c47

vstinner · 2025-05-01T16:26:01Z

Extract of the gdb backtrace:

#0  _PyObject_GC_UNTRACK (op=0x7fffdbc073d0) at ./Include/internal/pycore_gc.h:279
#1  listiter_dealloc (self=0x7fffdbc073d0) at Objects/listobject.c:3997
#2  0x00000000004f1a2e in _PyTrash_thread_destroy_chain (tstate=<optimized out>) at Objects/object.c:2957
#3  _Py_Dealloc (op=op@entry=0x7fffdbbf7f40) at Objects/object.c:3073
#4  0x00000000004aeda8 in Py_DECREF (op=0x7fffdbbf7f40) at ./Include/refcount.h:433
#5  framelocalsproxy_repr (self=0x7fffdefaf760) at Objects/frameobject.c:532
#6  framelocalsproxy_repr (self=0x7fffdefaf760) at Objects/frameobject.c:512
#7  0x00000000004f1f4a in PyObject_Repr (v=0x7fffdefaf760) at ./Include/object.h:270
...
#310188 0x00000000004f1f4a in PyObject_Repr (v=0x7fffdc742b40) at ./Include/object.h:270
#310189 PyObject_Repr (v=v@entry=0x7fffdc742b40) at Objects/object.c:753
#310190 0x00000000004aed74 in framelocalsproxy_repr (self=0x7fffdc72cf10) at Objects/frameobject.c:531
#310191 framelocalsproxy_repr (self=0x7fffdc72cf10) at Objects/frameobject.c:512
#310192 0x00000000004f1f4a in PyObject_Repr (v=0x7fffdc72cf10) at ./Include/object.h:270
#310193 PyObject_Repr (v=0x7fffdc72cf10) at Objects/object.c:753
#310194 0x0000000000564e2b in PyUnicodeWriter_WriteRepr (writer=writer@entry=0x7fffdc742af0, obj=<optimized out>) at Objects/unicodeobject.c:13969
#310195 0x00000000004d7bd7 in dict_repr_lock_held (self=0x7fffdc742ac0) at Objects/dictobject.c:3372
#310196 dict_repr (self=0x7fffdc742ac0) at Objects/dictobject.c:3401
#310197 0x00000000004f1f4a in PyObject_Repr (v=0x7fffdc742ac0) at ./Include/object.h:270
#310198 PyObject_Repr (v=v@entry=0x7fffdc742ac0) at Objects/object.c:753
#310199 0x00000000004aed74 in framelocalsproxy_repr (self=0x7fffdc72cf40) at Objects/frameobject.c:531
#310200 framelocalsproxy_repr (self=0x7fffdc72cf40) at Objects/frameobject.c:512
#310201 0x00000000004f1f4a in PyObject_Repr (v=0x7fffdc72cf40) at ./Include/object.h:270
...
#310486 0x000000000067bb72 in Py_RunMain () at Modules/main.c:767

GCC 9 and older don't have __has_builtin(), but have __builtin_frame_address() function.

ambv · 2025-05-01T17:03:06Z

Also happened randomly on my Windows 11 buildbot:
https://buildbot.python.org/#/builders/1088/builds/722/steps/5/logs/stdio

vstinner · 2025-05-01T17:07:49Z

Also happened randomly on my Windows 11 buildbot:
https://buildbot.python.org/#/builders/1088/builds/722/steps/5/logs/stdio

Oh strange. This bug looks more like #133199 : assertion failure in _PyObject_GC_UNTRACK().

test_repr_deep (test.test_frame.FrameLocalsProxyMappingTests.test_repr_deep) ...

R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Objects\listobject.c:3997: _PyObject_GC_UNTRACK: Assertion "_PyObject_GC_IS_TRACKED(((PyObject*)(op)))" failed: object not tracked by the garbage collector
Enable tracemalloc to get the memory block allocation traceback
object address  : 0000016FA81B1090
object refcount : 0
object type     : 00007FFB97E7EEB0
object type name: list_iterator
object repr     : <refcnt 0 at 0000016FA81B1090>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized
Warning -- Unraisable exception
Exception ignored in the internal traceback machinery:
Traceback (most recent call last):
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 143, in _print_exception_bltin
    return print_exception(exc, limit=BUILTIN_EXCEPTION_LIMIT, file=file, colorize=colorize)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 134, in print_exception
    te.print(file=file, chain=chain, colorize=colorize)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 1577, in print
    for line in self.format(chain=chain, colorize=colorize):
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 1514, in format
    yield from _ctx.emit(exc.format_exception_only(colorize=colorize))
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 1001, in emit
    for text in text_gen:
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 1268, in format_exception_only
    isinstance(self.__notes__, collections.abc.Sequence)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
RecursionError: Stack overflow (used 11719 kB)
Current thread 0x00000408 (most recent call first):
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\case.py", line 247 in handle
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\case.py", line 813 in assertRaises
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\mapping_tests.py", line 634 in test_repr_deep
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\case.py", line 615 in _callTestMethod
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\case.py", line 669 in run
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\case.py", line 725 in __call__
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\suite.py", line 122 in run
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\suite.py", line 84 in __call__
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\suite.py", line 122 in run
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\suite.py", line 84 in __call__
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\runner.py", line 259 in run
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 84 in _run_suite
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 42 in run_unittest
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 162 in test_func
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 118 in regrtest_runner
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 165 in _load_run_test
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 210 in _runtest_env_changed_exc
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 319 in _runtest
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 348 in run_single_test
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\worker.py", line 92 in worker_process
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\worker.py", line 127 in main
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\worker.py", line 131 in <module>
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\runpy.py", line 88 in _run_code
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\runpy.py", line 198 in _run_module_as_main
Extension modules: _testinternalcapi, _testcapi (total: 2)

GCC 9 and older don't have __has_builtin(), but have __builtin_frame_address() function.

vstinner · 2025-05-01T17:15:51Z

@ambv: Would you mind to open a separated issue for the Windows failure? It's a different root cause related to _PyObject_GC_UNTRACK().

encukou · 2025-05-02T08:19:58Z

Since the PR was merged, the s390x RHEL8 buildbot started failing: https://buildbot.python.org/#/builders/442/builds/9249

StanFromIreland · 2025-05-04T16:10:46Z

Can also reproduce (~25% of the time) on my machine, x86-64 Fedora 42.

vstinner · 2025-05-04T16:15:05Z

The crash or the GC untrack assertion error?

StanFromIreland · 2025-05-04T16:16:54Z

The assertion error, apologies, I should have specified.

vstinner · 2025-05-04T16:18:33Z

Can you please open a new issue for the assertion error?

vstinner · 2025-05-04T17:13:51Z

I can reproduce the crash ("Fatal Python error: Segmentation fault") on x86-64 Fedora 42:

make clean
./configure
make
./python -m test -v test_frame  # run multiple times until you get a crash

Output:

test_repr_deep (test.test_frame.FrameLocalsProxyMappingTests.test_repr_deep) ...

Fatal Python error: Segmentation fault
(...)

vstinner · 2025-05-04T17:18:07Z

@markshannon: I don't understand why _PyTrash_thread_destroy_chain() is called by _Py_Dealloc() whereas the call stack is very deep (150 000 Python frames / 300 000 C frames). Is it a bug in _Py_RecursionLimit_GetMargin()? Or is the limit badly configured?

Fix a random crash in test_frame.test_repr_deep() on x86-64.

vstinner · 2025-05-04T19:59:10Z

I wrote #133401 to fix test_frame random crash.

vstinner · 2025-05-04T20:26:18Z

I also saw the crash on "PPC64LE RHEL8 LTO + PGO 3.x" buildbot: https://buildbot.python.org/#/builders/458/builds/6121

test_repr_deep (test.test_frame.FrameLocalsProxyMappingTests.test_repr_deep) ... Fatal Python error: Segmentation fault

Current thread 0x00007fffb9b845e0 [python] (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/unittest/case.py", line 247 in handle
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/unittest/case.py", line 813 in assertRaises
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/test/mapping_tests.py", line 634 in test_repr_deep

vstinner · 2025-05-05T08:52:10Z

I also saw the crash on "s390x RHEL8 LTO + PGO 3.x" buildbot: https://buildbot.python.org/#/builders/442/builds/9283

test_repr_deep (test.test_frame.FrameLocalsProxyMappingTests.test_repr_deep) ... Fatal Python error: Segmentation fault

Current thread 0x000003ffaf3f7270 [python] (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel8-s390x.lto-pgo/build/Lib/unittest/case.py", line 247 in handle
  File "/home/buildbot/buildarea/3.x.cstratak-rhel8-s390x.lto-pgo/build/Lib/unittest/case.py", line 813 in assertRaises
  File "/home/buildbot/buildarea/3.x.cstratak-rhel8-s390x.lto-pgo/build/Lib/test/mapping_tests.py", line 634 in test_repr_deep
  ...

markshannon · 2025-05-05T08:53:36Z

@markshannon: I don't understand why _PyTrash_thread_destroy_chain() is called by _Py_Dealloc() whereas the call stack is very deep (150 000 Python frames / 300 000 C frames). Is it a bug in _Py_RecursionLimit_GetMargin()? Or is the limit badly configured?

That's a bit surprising.
If recursion is so deep that we are raising a recursion error, we definitely shouldn't be calling _PyTrash_thread_destroy_chain().

markshannon · 2025-05-05T11:18:40Z

It appears that _PyTrash_thread_destroy_chain() is being called correctly, during later handling of the overflow.

…H-133431)

vstinner · 2025-05-05T12:49:03Z

Fixed by #133431. Thanks @markshannon.

vstinner · 2025-05-05T15:25:47Z

I reopen the issue, sadly an assertion fails on x86 (32-bit): #133431 (comment)

Make sure trashcan pointer look mortal -- 32 bit

hugovk · 2025-05-06T07:11:59Z

Has the merge of #133450 fixed this?

markshannon · 2025-05-06T08:20:14Z

I believe so, yes.

vstinner added tests Tests in the Lib/test dir type-crash A hard crash of the interpreter, possibly with a core dump labels May 1, 2025

vstinner mentioned this issue May 1, 2025

gh-124715: Fix method_dealloc(): use PyObject_GC_UnTrack() #133199

Merged

vstinner added a commit to vstinner/cpython that referenced this issue May 1, 2025

pythongh-133261: Use __builtin_frame_address() on GCC 9 and older

f6ef2e3

GCC 9 and older don't have __has_builtin(), but have __builtin_frame_address() function.

bedevere-app bot mentioned this issue May 1, 2025

gh-133261: Use __builtin_frame_address() on GCC 9 and older #133269

Merged

vstinner added a commit that referenced this issue May 1, 2025

gh-133261: Use __builtin_frame_address() on GCC 9 and older (#133269)

d10bd81

GCC 9 and older don't have __has_builtin(), but have __builtin_frame_address() function.

emmatyping mentioned this issue May 3, 2025

gh-132983: Introduce _zstd bindings module #133027

Merged

4 tasks

StanFromIreland mentioned this issue May 4, 2025

test_frame.test_repr_deep() fails randomly on Windows11 Bigmem buildbot: _PyObject_GC_UNTRACK() assertion error #133384

Closed

vstinner added a commit to vstinner/cpython that referenced this issue May 4, 2025

pythongh-133261: Increase stack margin

b0bf8c6

Fix a random crash in test_frame.test_repr_deep() on x86-64.

vstinner added a commit to vstinner/cpython that referenced this issue May 4, 2025

pythongh-133261: Increase stack margin

c75cb64

Fix a random crash in test_frame.test_repr_deep() on x86-64.

vstinner added a commit to vstinner/cpython that referenced this issue May 4, 2025

pythongh-133261: Increase C stack margin

cc1c519

Fix a random crash in test_frame.test_repr_deep() on x86-64.

bedevere-app bot mentioned this issue May 4, 2025

gh-133261: Increase C stack margin #133401

Closed

vstinner added the release-blocker label May 4, 2025

ezio-melotti added this to Release and Deferred blockers 🚫 May 4, 2025

github-project-automation bot moved this to Todo in Release and Deferred blockers 🚫 May 4, 2025

vstinner changed the title ~~test_frame does crash randomly on AMD64 RHEL8 LTO 3.x~~ test_frame does crash randomly on Linux x86-64 May 4, 2025

vstinner changed the title ~~test_frame does crash randomly on Linux x86-64~~ test_frame does crash randomly on Linux (x86-64 and ppc64le) May 4, 2025

vstinner changed the title ~~test_frame does crash randomly on Linux (x86-64 and ppc64le)~~ test_frame does crash randomly on Linux (x86-64, ppc64le, s390x) May 5, 2025

bedevere-app bot mentioned this issue May 5, 2025

GH-133261: Make sure that the GC doesn't untrack objects in trashcan #133431

Merged

markshannon added a commit that referenced this issue May 5, 2025

GH-133261: Make sure that the GC doesn't untrack objects in trashcan (G…

f554237

…H-133431)

vstinner closed this as completed May 5, 2025

github-project-automation bot moved this from Todo to Done in Release and Deferred blockers 🚫 May 5, 2025

hugovk mentioned this issue May 5, 2025

gh-107006: Move threading.local docstring to docs #131840

Merged

vstinner reopened this May 5, 2025

github-project-automation bot moved this from Done to In Progress in Release and Deferred blockers 🚫 May 5, 2025

bedevere-app bot mentioned this issue May 5, 2025

GH-133261: Make sure trashcan pointers look mortal -- 32 bit #133450

Merged

ambv pushed a commit that referenced this issue May 5, 2025

GH-133261: Make sure trashcan pointers look mortal -- 32 bit (GH-133450)

d48616e

Make sure trashcan pointer look mortal -- 32 bit

hugovk closed this as completed May 6, 2025

github-project-automation bot moved this from In Progress to Done in Release and Deferred blockers 🚫 May 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_frame does crash randomly on Linux (x86-64, ppc64le, s390x) #133261

test_frame does crash randomly on Linux (x86-64, ppc64le, s390x) #133261

vstinner commented May 1, 2025 •

edited by bedevere-app bot

Loading

vstinner commented May 1, 2025

vstinner commented May 1, 2025

ambv commented May 1, 2025

vstinner commented May 1, 2025

vstinner commented May 1, 2025

encukou commented May 2, 2025

StanFromIreland commented May 4, 2025 •

edited

Loading

vstinner commented May 4, 2025

StanFromIreland commented May 4, 2025 •

edited

Loading

vstinner commented May 4, 2025

vstinner commented May 4, 2025

vstinner commented May 4, 2025

vstinner commented May 4, 2025

vstinner commented May 4, 2025

vstinner commented May 5, 2025

markshannon commented May 5, 2025

markshannon commented May 5, 2025

vstinner commented May 5, 2025

vstinner commented May 5, 2025

hugovk commented May 6, 2025

markshannon commented May 6, 2025

test_frame does crash randomly on Linux (x86-64, ppc64le, s390x) #133261

test_frame does crash randomly on Linux (x86-64, ppc64le, s390x) #133261

Comments

vstinner commented May 1, 2025 • edited by bedevere-app bot Loading

Crash report

Linked PRs

vstinner commented May 1, 2025

vstinner commented May 1, 2025

ambv commented May 1, 2025

vstinner commented May 1, 2025

vstinner commented May 1, 2025

encukou commented May 2, 2025

StanFromIreland commented May 4, 2025 • edited Loading

vstinner commented May 4, 2025

StanFromIreland commented May 4, 2025 • edited Loading

vstinner commented May 4, 2025

vstinner commented May 4, 2025

vstinner commented May 4, 2025

vstinner commented May 4, 2025

vstinner commented May 4, 2025

vstinner commented May 5, 2025

markshannon commented May 5, 2025

markshannon commented May 5, 2025

vstinner commented May 5, 2025

vstinner commented May 5, 2025

hugovk commented May 6, 2025

markshannon commented May 6, 2025

vstinner commented May 1, 2025 •

edited by bedevere-app bot

Loading

StanFromIreland commented May 4, 2025 •

edited

Loading

StanFromIreland commented May 4, 2025 •

edited

Loading