segmentation fault python macos #352
Comments
|
Yes. I briefly looked into this for the conda package, where the tests for the Python bindings failed due to this crash. However since I do not have a macOS system, I cannot really debug this issue.
I guess it would help to get a backtrace. That would mean that you have to compile the project and the Python bindings with Debug symbols, then run the Python interpreter in gdb or similar, reproduce this crash, and finally print the backtrace and paste it here. |
|
I am happy to help at the debugging process. Unfortunately I am not experienced in that field so I fear that you need to walk me a bit through this. What I did is compiling the library with the following command: gdb is not available for apple silicon so I used lldb, if there is a better alternative, please let me know. I hope that this helps. |
|
Building with If I force a crash via gdb -ex run --args python3 -c "import apriltag"which will give: Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7fadb3f in PyInit_apriltag () at [...]/apriltag/apriltag_pywrap.c:380
380 printf("NULL: %d\n", *(int*)NULL);
(gdb) bt
#0 0x00007ffff7fadb3f in PyInit_apriltag () at [...]/apriltag/apriltag_pywrap.c:380
#1 0x00000000006a9881 in _PyImport_LoadDynamicModuleWithSpec (spec=0x7ffff74349e0, fp=<optimized out>) at ../Python/importdl.c:169
#2 0x00000000006a8fd2 in _imp_create_dynamic_impl (module=<optimized out>, file=0x0, spec=0x7ffff74349e0) at ../Python/import.c:3775
#3 _imp_create_dynamic (module=<optimized out>, args=<optimized out>, nargs=<optimized out>) at ../Python/clinic/import.c.h:506
#4 0x0000000000582067 in cfunction_vectorcall_FASTCALL (func=0x7ffff75972e0, args=0x7ffff75fc928, nargsf=<optimized out>, kwnames=<optimized out>)
at ../Include/cpython/methodobject.h:50
#5 0x00000000005db336 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/bytecodes.c:3254
#6 0x0000000000549ae7 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffffd500, callable=0x7ffff75a4040, tstate=0xba6048 <_PyRuntime+459656>)
at ../Include/internal/pycore_call.h:92
#7 object_vacall (tstate=tstate@entry=0xba6048 <_PyRuntime+459656>, base=<optimized out>, callable=0x7ffff75a4040, vargs=0x7fffffffd588) at ../Objects/call.c:850
#8 0x000000000054b373 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at ../Objects/call.c:911
#9 0x00000000005fda35 in import_find_and_load (abs_name=0x7ffff743de30, tstate=0xba6048 <_PyRuntime+459656>) at ../Python/import.c:2779
#10 PyImport_ImportModuleLevelObject (name=name@entry=0x7ffff743de30, globals=<optimized out>, locals=locals@entry=0x7ffff75f9e80, fromlist=fromlist@entry=0xa408a0 <_Py_NoneStruct>,
level=0) at ../Python/import.c:2862
#11 0x00000000005dc40f in import_name (level=0xb36988 <_PyRuntime+3272>, fromlist=0xa408a0 <_Py_NoneStruct>, name=0x7ffff743de30, frame=<optimized out>, tstate=<optimized out>)
at ../Python/ceval.c:2482
#12 _PyEval_EvalFrameDefault (tstate=tstate@entry=0xba6048 <_PyRuntime+459656>, frame=<optimized out>, frame@entry=0x7ffff7fb2020, throwflag=throwflag@entry=0)
at Python/bytecodes.c:2135
#13 0x00000000005d560b in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb2020, tstate=0xba6048 <_PyRuntime+459656>) at ../Include/internal/pycore_ceval.h:89
#14 _PyEval_Vector (kwnames=0x0, argcount=0, args=0x0, locals=0x7ffff75f9e80, func=0x7ffff744d3a0, tstate=0xba6048 <_PyRuntime+459656>) at ../Python/ceval.c:1683
#15 PyEval_EvalCode (co=co@entry=0x7ffff748cfa0, globals=globals@entry=0x7ffff75f9e80, locals=locals@entry=0x7ffff75f9e80) at ../Python/ceval.c:578
#16 0x00000000006086f3 in run_eval_code_obj (locals=0x7ffff75f9e80, globals=0x7ffff75f9e80, co=0x7ffff748cfa0, tstate=0xba6048 <_PyRuntime+459656>) at ../Python/pythonrun.c:1722
#17 run_mod (arena=0x7ffff751be50, flags=0x7ffff751be50, locals=0x7ffff75f9e80, globals=0x7ffff75f9e80, filename=<optimized out>, mod=<optimized out>) at ../Python/pythonrun.c:1743
#18 PyRun_StringFlags (str=str@entry=0x7ffff75fa050 "import apriltag\n", start=start@entry=257, globals=0x7ffff75f9e80, locals=0x7ffff75f9e80, flags=flags@entry=0x7fffffffd9c0)
at ../Python/pythonrun.c:1618
#19 0x00000000006b40ee in PyRun_SimpleStringFlags (command=0x7ffff75fa050 "import apriltag\n", flags=flags@entry=0x7fffffffd9c0) at ../Python/pythonrun.c:480
#20 0x00000000006bce01 in pymain_run_command (command=<optimized out>) at ../Modules/main.c:255
#21 pymain_run_python (exitcode=0x7fffffffd98c) at ../Modules/main.c:620
#22 Py_RunMain () at ../Modules/main.c:709
#23 0x00000000006bc81d in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:763
#24 0x00007ffff7c2a1ca in __libc_start_call_main (main=main@entry=0x518880 <main>, argc=argc@entry=3, argv=argv@entry=0x7fffffffdbd8) at ../sysdeps/nptl/libc_start_call_main.h:58
#25 0x00007ffff7c2a28b in __libc_start_main_impl (main=0x518880 <main>, argc=3, argv=0x7fffffffdbd8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
stack_end=0x7fffffffdbc8) at ../csu/libc-start.c:360
#26 0x0000000000657ca5 in _start ()Line 380 is then exactly the line where I added the NULL-pointer dereference. |
|
The line Since you do not seem to be able to use a debugger for now to debug this, could you simply add a print statement like such: if (PyType_Ready(&apriltagType) < 0) {
printf("PyType_Ready error!\n"); fflush(stdout);
return NULL;
}and check if it prints on the screen? |
|
I added a test for This might just be related to a weird Python setup with a mixup of versions from different sources. Your initial report shows that you are using |
|
I recompiled it and judging from |
|
The different Python version where used in different environments. But always compiled against the correct version of the environment. I will definitely check with the standard version although I need it to work in the conda environment. |
Did you add a print statement? At least now you get the line numbers. |
|
Interestingly, it does work like a charm when I am building outside of the conda environment but still with the brew python. But I think it crashes at |
That sounds definitely like a mixup of environments.
Which version of the code are you on? On the current master, line 375 is Lines 374 to 375 in 786ad11 If this line appear in the backtrace, a print before this, e.g.: if (PyType_Ready(&apriltagType) < 0) { // 374
printf("PyType_Ready error!\n"); fflush(stdout); // 375
return NULL; // 376
}should have been shown in the terminal. |
|
But shouldn't it matter what environments you have, as long as you choose the right ones when building the library? I am using the current master despite adding the lines that you asked me to. It is possible that some auto-linting misaligned some of the stuff. Now I added a print statement before the if statement. So the line count is off by one: |
No. The build and runtime environments have to be the same. Linking against a different library than you use later can cause different kinds of ABI incompatibilities.
Well, ideally your backtraces match with the source code of the repo. Otherwise, it's hard to use it to debug what is going on. If the crash is indeed inside If you can reduce the crash in the CI, I can have a look at it. But other than this, I recommend that you fix your Python environment. |
|
To clarify what I meant was using the same build and runtime environments. Meaning that when I specify the and using this specify environment during the run then it should not matter that there are other environments installed on the machine right? |
|
I am not sure how this works with the mixed homebrew and mambaforge environment. At least on Linux, if you have set the environment, e.g. conda, correctly, then you also don't need to point to absolute paths for Python manually since CMake will find the Since your environment is named |
|
change CMakeLists.txt: Line 182 in 724a7d8 to and it works on macos with conda. |
|
@clysto Do you have any data on why removing The CI builds and imports the |
|
I have no idea why this happens, but this did solve the issue for me as well. |
|
@clysto @jgibson2 Can you provide details about the build environment and a backtrace in gdb (or any other debugger)? This looks like it is caused by a mixup of the Python environment (see the previous comments in this issue) and works outside of conda. And a crash inside |
|
From what I understand, Python3_add_library() already handles linking to the appropriate Python interpreter and runtime when building a Python module. Explicitly linking Python3::Python again—especially when using a Conda-based Python environment—may cause symbol duplication or conflicts, particularly with the Python runtime already loaded by the interpreter at runtime. This redundancy can lead to undefined behavior, including segmentation faults, as observed. Therefore, removing the Python3::Python target from the target_link_libraries() call is not only sufficient but also necessary in this context. |
|
So per se, linking @clysto Can you show that |
|
I believe this is due to the difference in how Python’s dynamic libraries are installed in the Conda environment versus the system Python on macOS. In any case, if the code does not need to embed the Python interpreter, I think there’s no need to link against the Python library—using the header files alone should be sufficient. |
#define PY_SSIZE_T_CLEAN
#include <Python.h>
static PyObject *add(PyObject *self, PyObject *args) {
int a, b;
if (!PyArg_ParseTuple(args, "ii", &a, &b)) {
return NULL;
}
return PyLong_FromLong(a + b);
}
static PyMethodDef all_methods[] = {{"add", add, METH_VARARGS, ""},
{NULL, NULL, 0, NULL}};
static struct PyModuleDef demo_definition = {
PyModuleDef_HEAD_INIT, "demo", "A demo Python module.", -1, all_methods};
PyMODINIT_FUNC PyInit_demo(void) {
return PyModule_Create(&demo_definition);
}This is a minimal reproducible example: when compiled in a Conda environment, it results in a segmentation fault, whereas compiling with the system Python does not cause any issues. If libpython is not linked, the minimal example works correctly in both environments. I haven’t fully understood why this happens yet, but in many cases, linking against libpython might not be necessary. Do you have any better ideas or suggestions? |
The CMake target
Thanks for the minimal example. This shows at least that this is not specific to the apriltag library. It's still not clear what is happening or why But if this workaround fixes an issue and does not affect the behaviour of the rest, I am willing to review and marge a PR. If you do so, document the workaround with a comment in the |
fix segmentation fault in conda env #352
Hello,
I am trying to build the Python bindings for macOS. The building process works but when trying to import the library, there is always a segmentation fault. Using Python 3.12.6 with a conda environment. I was able to build the duckietown bindings but I receive a lot of
Error, more than one new minimum found.errors with them. I did not find an actual solution for this and it seems that this error does not exists with the offical bindings.If I should provide more system information please let me know.
The text was updated successfully, but these errors were encountered: