Heap buffer overflow reported by ASAN

[ngoldbaum]

If you build an ASAN-instrumented version of CPython by passing e.g. --with-address-sanitizer to the CPython configure script or setting CONFIGURE_OPTS="--with-address-sanitizer" while building CPython with pyenv and then compile python-cffi with ASAN support like this, from the root of the repository:

CFLAGS='-fsanitize=address' python -m pip install .

And run the following test command:

pytest src/c/test_c.py -sv

You will eventually see a heap buffer overflow report while running test_from_buffer_types:

src/c/test_c.py::test_from_buffer_types =================================================================
==790==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200015ce6c at pc 0x000106a585e8 bp 0x00016b286a40 sp 0x00016b2861f0
READ of size 4 at 0x60200015ce6c thread T0
    #0 0x000106a585e4 in __asan_memcpy+0x3d4 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x505e4)
    #1 0x00010a803aa0 in read_raw_signed_data _cffi_backend.c:934
    #2 0x00010a803208 in convert_to_object _cffi_backend.c:1118
    #3 0x00010a827174 in cdata_subscript _cffi_backend.c:2736
    #4 0x000105d19f90 in PyObject_GetItem abstract.c:158
    #5 0x000106057ecc in _PyEval_EvalFrameDefault generated_cases.c.h:446
    #6 0x000105d5c670 in _PyObject_VectorcallDictTstate call.c:146
    #7 0x000105d5f108 in _PyObject_Call_Prepend call.c:504
    #8 0x000105eb2380 in slot_tp_call typeobject.c:9565
    #9 0x000105d5cacc in _PyObject_MakeTpCall call.c:242
    #10 0x00010605386c in _PyEval_EvalFrameDefault generated_cases.c.h:1502
    #11 0x000105d5c670 in _PyObject_VectorcallDictTstate call.c:146
    #12 0x000105d5f108 in _PyObject_Call_Prepend call.c:504
    #13 0x000105eb2380 in slot_tp_call typeobject.c:9565
    #14 0x000105d5e4e4 in _PyObject_Call call.c:361
    #15 0x00010605269c in _PyEval_EvalFrameDefault generated_cases.c.h:1355
    #16 0x000105d5c670 in _PyObject_VectorcallDictTstate call.c:146
    #17 0x000105d5f108 in _PyObject_Call_Prepend call.c:504
    #18 0x000105eb2380 in slot_tp_call typeobject.c:9565
    #19 0x000105d5cacc in _PyObject_MakeTpCall call.c:242
    #20 0x00010605386c in _PyEval_EvalFrameDefault generated_cases.c.h:1502
    #21 0x000105d5c670 in _PyObject_VectorcallDictTstate call.c:146
    #22 0x000105d5f108 in _PyObject_Call_Prepend call.c:504
    #23 0x000105eb2380 in slot_tp_call typeobject.c:9565
    #24 0x000105d5cacc in _PyObject_MakeTpCall call.c:242
    #25 0x00010605386c in _PyEval_EvalFrameDefault generated_cases.c.h:1502
    #26 0x000105d5c670 in _PyObject_VectorcallDictTstate call.c:146
    #27 0x000105d5f108 in _PyObject_Call_Prepend call.c:504
    #28 0x000105eb2380 in slot_tp_call typeobject.c:9565
    #29 0x000105d5cacc in _PyObject_MakeTpCall call.c:242
    #30 0x00010605386c in _PyEval_EvalFrameDefault generated_cases.c.h:1502
    #31 0x00010603f57c in PyEval_EvalCode ceval.c:604
    #32 0x00010615d678 in run_eval_code_obj pythonrun.c:1381
    #33 0x00010615cf3c in run_mod pythonrun.c:1466
    #34 0x0001061581a8 in _PyRun_SimpleFileObject pythonrun.c:517
    #35 0x000106157764 in _PyRun_AnyFileObject pythonrun.c:77
    #36 0x0001061af738 in pymain_run_file main.c:429
    #37 0x0001061ae150 in Py_RunMain main.c:775
    #38 0x0001061ae8c8 in pymain_main main.c:805
    #39 0x0001061aebcc in Py_BytesMain main.c:829
    #40 0x00019dc92b94 in start+0x17b8 (dyld:arm64+0xfffffffffff3ab94)

0x60200015ce6c is located 4 bytes before 16-byte region [0x60200015ce70,0x60200015ce80)
allocated by thread T0 here:
    #0 0x000106a5b6d0 in realloc+0x78 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x536d0)
    #1 0x000105d3075c in PyByteArray_Resize bytearrayobject.c:234
    #2 0x000105d322b8 in bytearray___init__ bytearrayobject.c.h:98
    #3 0x000105ea0fb8 in type_call typeobject.c:1997
    #4 0x000105d5cacc in _PyObject_MakeTpCall call.c:242
    #5 0x0001060510c0 in _PyEval_EvalFrameDefault generated_cases.c.h:813
    #6 0x000105d5c670 in _PyObject_VectorcallDictTstate call.c:146
    #7 0x000105d5f108 in _PyObject_Call_Prepend call.c:504
    #8 0x000105eb2380 in slot_tp_call typeobject.c:9565
    #9 0x000105d5cacc in _PyObject_MakeTpCall call.c:242
    #10 0x00010605386c in _PyEval_EvalFrameDefault generated_cases.c.h:1502
    #11 0x000105d5c670 in _PyObject_VectorcallDictTstate call.c:146
    #12 0x000105d5f108 in _PyObject_Call_Prepend call.c:504
    #13 0x000105eb2380 in slot_tp_call typeobject.c:9565
    #14 0x000105d5e4e4 in _PyObject_Call call.c:361
    #15 0x00010605269c in _PyEval_EvalFrameDefault generated_cases.c.h:1355
    #16 0x000105d5c670 in _PyObject_VectorcallDictTstate call.c:146
    #17 0x000105d5f108 in _PyObject_Call_Prepend call.c:504
    #18 0x000105eb2380 in slot_tp_call typeobject.c:9565
    #19 0x000105d5cacc in _PyObject_MakeTpCall call.c:242
    #20 0x00010605386c in _PyEval_EvalFrameDefault generated_cases.c.h:1502
    #21 0x000105d5c670 in _PyObject_VectorcallDictTstate call.c:146
    #22 0x000105d5f108 in _PyObject_Call_Prepend call.c:504
    #23 0x000105eb2380 in slot_tp_call typeobject.c:9565
    #24 0x000105d5cacc in _PyObject_MakeTpCall call.c:242
    #25 0x00010605386c in _PyEval_EvalFrameDefault generated_cases.c.h:1502
    #26 0x000105d5c670 in _PyObject_VectorcallDictTstate call.c:146
    #27 0x000105d5f108 in _PyObject_Call_Prepend call.c:504
    #28 0x000105eb2380 in slot_tp_call typeobject.c:9565
    #29 0x000105d5cacc in _PyObject_MakeTpCall call.c:242

SUMMARY: AddressSanitizer: heap-buffer-overflow _cffi_backend.c:934 in read_raw_signed_data
Shadow bytes around the buggy address:
  0x60200015cb80: fa fa fd fd fa fa fd fa fa fa fd fa fa fa fd fa
  0x60200015cc00: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fd
  0x60200015cc80: fa fa fd fd fa fa fd fd fa fa fd fa fa fa fd fa
  0x60200015cd00: fa fa fd fa fa fa fd fa fa fa fd fa fa fa 00 fa
  0x60200015cd80: fa fa fd fa fa fa 00 fa fa fa fd fa fa fa 00 fa
=>0x60200015ce00: fa fa fd fa fa fa 00 fa fa fa 00 fa fa[fa]00 00
  0x60200015ce80: fa fa fd fd fa fa fa fa fa fa fa fa fa fa fa fa
  0x60200015cf00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x60200015cf80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x60200015d000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x60200015d080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==790==ABORTING
Fatal Python error: Aborted

[arigo]

Some lines have the comment "hopefully does not crash, but doesn't raise an exception" (repeated several times). I think the goal was to test a case where CFFI is not able to detect out-of-bounds access. The test does the access anyway and checks that indeed no exception was raised. In corner cases the access could crash instead, but that is very unlikely in practice. But of course it's exactly the case that a C-level debugging tool would detect and rightfully complain about. But I don't know how to fix the test without weakening it; unless somebody has got a suggestion, it's a "won't fix" from me.

[ngoldbaum]

There's probably a way to detect that either Python or CFFI were compiled with ASAN and skip the test in that case.

For my purposes I can skip the test when I'm debugging locally.