Python-3.13-iOS-support.b4 crash in -[iOSTestbedTests testPython]

[mingchen]

Describe the bug

Hello,

This is great project, thanks for making it.

OS: Mac OS 14.7.1 (23H222)
Xcode: 16.2
Python framework: 3.13-b4

Steps to reproduce

1. Download python 3.13-b4 from https://github.com/beeware/Python-Apple-support/releases/tag/3.13-b4
2. Launch iOSTestbed.xcodeproj
3. Run unit tests on iOS 18.3.1 Simulator
4. Unit test crashed. see blow

Thread 1 Queue : com.apple.main-thread (serial)
#0	0x0000000104da09a4 in __kill ()
#1	0x00000001066783b0 in os_kill ()
#2	0x00000001065110dc in cfunction_vectorcall_FASTCALL ()
#3	0x00000001064c0348 in PyObject_Vectorcall ()
#4	0x00000001065e9f38 in _PyEval_EvalFrameDefault ()
#5	0x00000001064c2afc in method_vectorcall ()
#6	0x00000001065e9b00 in _PyEval_EvalFrameDefault ()
#7	0x00000001064bfa8c in _PyObject_VectorcallDictTstate ()
#8	0x00000001064c09b4 in _PyObject_Call_Prepend ()
#9	0x000000010654b64c in slot_tp_call ()
#10	0x00000001064bfc4c in _PyObject_MakeTpCall ()
#11	0x00000001065e9254 in _PyEval_EvalFrameDefault ()
#12	0x00000001064c2afc in method_vectorcall ()
#13	0x00000001065e9b00 in _PyEval_EvalFrameDefault ()
#14	0x00000001064bfa8c in _PyObject_VectorcallDictTstate ()
#15	0x00000001064c09b4 in _PyObject_Call_Prepend ()
#16	0x000000010654b64c in slot_tp_call ()
#17	0x00000001064bfc4c in _PyObject_MakeTpCall ()
#18	0x00000001065e9254 in _PyEval_EvalFrameDefault ()
#19	0x00000001064c2afc in method_vectorcall ()
#20	0x00000001065e9b00 in _PyEval_EvalFrameDefault ()
#21	0x00000001064bfa8c in _PyObject_VectorcallDictTstate ()
#22	0x00000001064c09b4 in _PyObject_Call_Prepend ()
#23	0x000000010654b64c in slot_tp_call ()
#24	0x00000001064bfc4c in _PyObject_MakeTpCall ()
#25	0x00000001065e9254 in _PyEval_EvalFrameDefault ()
#26	0x00000001065e4ab0 in PyEval_EvalCode ()
#27	0x00000001065e1b6c in builtin_exec ()
#28	0x00000001065111a0 in cfunction_vectorcall_FASTCALL_KEYWORDS ()
#29	0x00000001064c0348 in PyObject_Vectorcall ()
#30	0x00000001065e9f38 in _PyEval_EvalFrameDefault ()
#31	0x000000010666e0d0 in pymain_run_module ()
#32	0x000000010666d72c in Py_RunMain ()
#33	0x0000000105cb29b8 in -[iOSTestbedTests testPython] at Downloads/Python-3.13-iOS-support.b4/testbed/iOSTestbedTests/iOSTestbedTests.m:153
#34	0x00000001804bfce0 in __invoking___ ()
#35	0x00000001804bcc54 in -[NSInvocation invoke] ()
#36	0x0000000105d8eac4 in +[XCTFailableInvocation invokeStandardConventionInvocation:completion:] ()
#37	0x0000000105d8ea7c in __90+[XCTFailableInvocation invokeInvocation:withTestMethodConvention:lastObservedErrorIssue:]_block_invoke_3 ()
#38	0x0000000105d8e2e8 in __81+[XCTFailableInvocation invokeWithAsynchronousWait:lastObservedErrorIssue:block:]_block_invoke ()
#39	0x0000000105d744d4 in __49+[XCTSwiftErrorObservation observeErrorsInBlock:]_block_invoke ()
#40	0x0000000106d8a9a4 in reabstraction thunk helper from @callee_unowned @convention(block) () -> () to @escaping @callee_guaranteed () -> () ()
#41	0x0000000106d8a804 in closure #1 () -> () in static __C.XCTSwiftErrorObservation._observeErrors(in: () -> ()) -> () -> Swift.Optional<XCTest.XCTIssue> ()
#42	0x0000000106d8aae0 in partial apply forwarder for closure #1 () -> () in static __C.XCTSwiftErrorObservation._observeErrors(in: () -> ()) -> () -> Swift.Optional<XCTest.XCTIssue> ()
#43	0x0000000249593e20 in Swift.TaskLocal.withValue<τ_0_0>(_: τ_0_0, operation: () throws -> τ_1_0, file: Swift.String, line: Swift.UInt) throws -> τ_1_0 ()
#44	0x0000000106d8a640 in static __C.XCTSwiftErrorObservation._observeErrors(in: () -> ()) -> () -> Swift.Optional<XCTest.XCTIssue> ()
#45	0x0000000106d8a918 in @objc static __C.XCTSwiftErrorObservation._observeErrors(in: () -> ()) -> () -> Swift.Optional<XCTest.XCTIssue> ()
#46	0x0000000105d743d0 in +[XCTSwiftErrorObservation observeErrorsInBlock:] ()
#47	0x0000000105d8e0c8 in +[XCTFailableInvocation invokeWithAsynchronousWait:lastObservedErrorIssue:block:] ()
#48	0x0000000105d8e7cc in +[XCTFailableInvocation invokeInvocation:withTestMethodConvention:lastObservedErrorIssue:] ()
#49	0x0000000105d8eb50 in +[XCTFailableInvocation invokeInvocation:lastObservedErrorIssue:] ()
#50	0x0000000105d4b268 in __24-[XCTestCase invokeTest]_block_invoke.80 ()
#51	0x0000000105d506dc in -[XCTestCase(XCTIssueHandling) _caughtUnhandledDeveloperExceptionPermittingControlFlowInterruptions:caughtInterruptionException:whileExecutingBlock:] ()
#52	0x0000000105d4adf8 in -[XCTestCase invokeTest] ()
#53	0x0000000105d4c86c in __26-[XCTestCase performTest:]_block_invoke.118 ()
#54	0x0000000105d506dc in -[XCTestCase(XCTIssueHandling) _caughtUnhandledDeveloperExceptionPermittingControlFlowInterruptions:caughtInterruptionException:whileExecutingBlock:] ()
#55	0x0000000105d4c2ac in __26-[XCTestCase performTest:]_block_invoke.108 ()
#56	0x0000000105d2b7a4 in +[XCTContext _runInChildOfContext:forTestCase:markAsReportingBase:block:] ()
#57	0x0000000105d2b6a4 in +[XCTContext runInContextForTestCase:markAsReportingBase:block:] ()
#58	0x0000000105d4bd58 in -[XCTestCase performTest:] ()
#59	0x0000000105d47fac in -[XCTest runTest] ()
#60	0x0000000105d56cec in -[XCTestSuite runTestBasedOnRepetitionPolicy:testRun:] ()
#61	0x0000000105d56b70 in __27-[XCTestSuite performTest:]_block_invoke ()
#62	0x0000000105d565f8 in __59-[XCTestSuite _performProtectedSectionForTest:testSection:]_block_invoke ()
#63	0x0000000105d2b7a4 in +[XCTContext _runInChildOfContext:forTestCase:markAsReportingBase:block:] ()
#64	0x0000000105d2b6a4 in +[XCTContext runInContextForTestCase:markAsReportingBase:block:] ()
#65	0x0000000105d56574 in -[XCTestSuite _performProtectedSectionForTest:testSection:] ()
#66	0x0000000105d5681c in -[XCTestSuite performTest:] ()
#67	0x0000000105d47fac in -[XCTest runTest] ()
#68	0x0000000105d7586c in __89-[XCTTestRunSession executeTestsWithIdentifiers:skippingTestsWithIdentifiers:completion:]_block_invoke ()
#69	0x0000000105d2b7a4 in +[XCTContext _runInChildOfContext:forTestCase:markAsReportingBase:block:] ()
#70	0x0000000105d2b6a4 in +[XCTContext runInContextForTestCase:markAsReportingBase:block:] ()
#71	0x0000000105d7558c in -[XCTTestRunSession executeTestsWithIdentifiers:skippingTestsWithIdentifiers:completion:] ()
#72	0x0000000105d6959c in __103-[XCTExecutionWorker executeTestIdentifiers:skippingTestIdentifiers:completionHandler:completionQueue:]_block_invoke_2 ()
#73	0x0000000105d69864 in __XCTAsyncEnumerateWithWaiter_block_invoke ()
#74	0x0000000105d699fc in __XCTAsyncEnumerate_block_invoke.98 ()
#75	0x0000000105d68510 in XCTAsyncEnumerateWithWaiter ()
#76	0x0000000105d69478 in __103-[XCTExecutionWorker executeTestIdentifiers:skippingTestIdentifiers:completionHandler:completionQueue:]_block_invoke ()
#77	0x0000000105d687c4 in -[XCTExecutionWorker runWithError:] ()
#78	0x0000000105d66808 in __25-[XCTestDriver _runTests]_block_invoke.200 ()
#79	0x0000000105d7a5f8 in -[XCTestObservationCenter _observeTestExecutionForTestBundle:inBlock:] ()
#80	0x0000000105d661f4 in -[XCTestDriver _runTests] ()
#81	0x0000000105d67f90 in _XCTestMain ()
#82	0x0000000104d17168 in RunTestsFromRunLoop ()
#83	0x00000001803e5a98 in __CFMachPortPerform ()
#84	0x000000018041d9ec in __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ ()
#85	0x000000018041cfb4 in __CFRunLoopDoSource1 ()
#86	0x000000018041740c in __CFRunLoopRun ()
#87	0x0000000180416704 in CFRunLoopRunSpecific ()
#88	0x0000000190604b10 in GSEventRunModal ()
#89	0x0000000185b39180 in -[UIApplication _run] ()
#90	0x0000000185b3d378 in UIApplicationMain ()
#91	0x0000000104c6e110 in main at Downloads/Python-3.13-iOS-support.b4/testbed/iOSTestbed/main.m:14
#92	0x0000000104e75410 in start_sim ()
#93	0x0000000104fd6154 in start ()

Expected behavior

Unit tests pass in iOSTestbedTests.

Screenshots

No response

Environment

• Operating System: Mac OS 14.7.1 (23H222)
• Python version: 13.3-b4
• Software versions:
  • Briefcase:
  • Toga:
  • ...

Logs

Xcode Console logs:

...
== CPU count: 12
== encodings: locale=US-ASCII FS=utf-8
== resources: all

Using random seed: 664587721
0:00:00 load avg: 95.76 Run 480 tests sequentially in a single process
0:00:00 load avg: 95.76 [  1/480] test.test_asyncio.test_base_events
0:00:01 load avg: 94.18 [  2/480] test.test_asyncio.test_buffered_proto
0:00:01 load avg: 94.18 [  3/480] test.test_asyncio.test_context
0:00:01 load avg: 94.18 [  4/480] test.test_asyncio.test_eager_task_factory
0:00:06 load avg: 95.60 [  5/480] test.test_asyncio.test_events
(lldb) 

Additional context

No response

[mingchen]

I also tested https://github.com/beeware/Python-Apple-support/releases/tag/3.13-b5 have the exact same crash.

[freakboy3742]

Thanks for the report.

The good news is that this isn't a crash - it's Xcode handling a completely expected signal (SIGINT) that the CPython test suite raises as part of testing that the interpreter is working as expected. CPython code involves using (and catching) signals, and so the test suite needs to raise SIGINT (and some other signals) as part of a normal testing run.

Unfortunately, the default behavior of Xcode is to insert a breakpoint whenever SIGINT (and a couple of other signals) are raised. When you hit one (as you have in the provided screenshots), the debugger should have the PLAY/PAUSE button enabled; you should be able to hit "PLAY" when you hit this signal, and the test suite will keep running... until it hits the next signal (the test suite raises a lot of them).

This is a behavior that can be disabled... manually. Unfortunately, there's no way (that I'm aware of) for us to ship a configuration that disables these breakpoints by default. The process for manually disabling these signals can be found in the iOS README for the CPython repo.

You can also avoid the breakpoints by running the test suite from the command line (python3.X testbed run -- test). When you use the command-line interface to run an Xcode test suite, there's no breakpoint interface, so the tests don't halt.

[mingchen]

Thanks for the explanation @freakboy3742.

After this change, I was able to run tests. Looks like most of tests are passed.

69 tests skipped:
    test.test_asyncio.test_subprocess
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test.test_gdb.test_backtrace
    test.test_gdb.test_cfunction test.test_gdb.test_cfunction_full
    test.test_gdb.test_misc test.test_gdb.test_pretty_print
    test.test_multiprocessing_fork.test_manager
    test.test_multiprocessing_fork.test_misc
    test.test_multiprocessing_fork.test_processes
    test.test_multiprocessing_fork.test_threads
    test.test_multiprocessing_forkserver.test_manager
    test.test_multiprocessing_forkserver.test_misc
    test.test_multiprocessing_forkserver.test_processes
    test.test_multiprocessing_forkserver.test_threads
    test.test_multiprocessing_spawn.test_manager
    test.test_multiprocessing_spawn.test_misc
    test.test_multiprocessing_spawn.test_processes
    test.test_multiprocessing_spawn.test_threads test_android
    test_argparse test_asdl_parser test_clinic test_cmd_line
    test_curses test_dbm_gnu test_devpoll test_dtrace test_embed
    test_epoll test_faulthandler test_file_eintr test_fork1
    test_free_threading test_generated_cases test_getopt test_grp
    test_idle test_ioctl test_launcher test_msvcrt
    test_multiprocessing_main_handling test_optparse
    test_perf_profiler test_perfmaps test_pty test_pwd test_pyrepl
    test_readline test_regrtest test_repl test_startfile
    test_subprocess test_syslog test_tcl test_tkinter test_tools
    test_ttk test_ttk_textonly test_turtle test_venv test_wait3
    test_wait4 test_winapi test_winconsoleio test_winreg test_winsound
    test_wmi

1 test skipped (resource denied):
    test_zipfile64

2 re-run tests:
    test.test_pydoc.test_pydoc test__colorize

2 tests failed:
    test.test_pydoc.test_pydoc test__colorize

408 tests OK.

Total duration: 12 min 18 sec
Total tests: run=39,709 failures=22 skipped=3,190
Total test files: run=481/480 failed=2 skipped=69 resource_denied=1 rerun=2

Attached full Xcode log: pythonframetest.log

[freakboy3742]

Those failures are both expected.

The test__colorise is failure is caused by (python/cpython#130293); the patches to address it haven't been merged into this repo yet (I'm currently working on updates to the support packages).

The test_pydoc test is a failure because we strip pyc files from our distribution artefact to minimise file size.