Skip to content

implicit declaration of function ‘_PyLong_Frexp’; did you mean ‘_PyLong_New’? [-Wimplicit-function-declaration] #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gothicVI opened this issue Dec 26, 2024 · 15 comments

Comments

@gothicVI
Copy link

Hi,

new python 3.13 environment and the installation fails with:

$ gcc --version
gcc (GCC) 14.2.1 20240910
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ pip3 install guppy3 
Collecting guppy3
  Downloading guppy3-3.1.4.post1.tar.gz (344 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: guppy3
  Building wheel for guppy3 (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [102 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-313/guppy
      copying guppy/__init__.py -> build/lib.linux-x86_64-cpython-313/guppy
      copying guppy/_version.py -> build/lib.linux-x86_64-cpython-313/guppy
      creating build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/Cat.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/Code.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/Descriptor.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/FSA.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/Glue.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/Help.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/IterPermute.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/KanExtension.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/KnuthBendix.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/RE.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/RE_Rect.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/__init__.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/cmd.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/etc.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/textView.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/tkcursors.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      copying guppy/etc/xterm.py -> build/lib.linux-x86_64-cpython-313/guppy/etc
      creating build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/Document.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/DottedTree.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/Exceptions.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/FileIO.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/Filer.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/Gsml.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/Help.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/Html.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/Latex.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/Main.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/SpecNodes.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/Tester.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/Text.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/XHTML.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      copying guppy/gsl/__init__.py -> build/lib.linux-x86_64-cpython-313/guppy/gsl
      creating build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Classifiers.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Console.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Doc.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/ImpSet.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Monitor.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/OutputHandling.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Part.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Path.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Prof.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/RM.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/RefPat.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Remote.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/RemoteConstants.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Spec.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Target.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/UniSet.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/Use.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/View.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/__init__.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      copying guppy/heapy/pbhelp.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy
      creating build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/__init__.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/support.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_Classifiers.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_ER.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_OutputHandling.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_Part.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_Path.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_RefPat.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_RetaGraph.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_Spec.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_UniSet.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_View.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_all.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_dependencies.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_gsl.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_heapyc.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_menuleak.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      copying guppy/heapy/test/test_sf.py -> build/lib.linux-x86_64-cpython-313/guppy/heapy/test
      creating build/lib.linux-x86_64-cpython-313/guppy/sets
      copying guppy/sets/__init__.py -> build/lib.linux-x86_64-cpython-313/guppy/sets
      copying guppy/sets/test.py -> build/lib.linux-x86_64-cpython-313/guppy/sets
      running build_ext
      building 'guppy.sets.setsc' extension
      creating build/temp.linux-x86_64-cpython-313/src/sets
      gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -fPIC -I/usr/include/python3.13 -c src/sets/bitset.c -o build/temp.linux-x86_64-cpython-313/src/sets/bitset.o
      src/sets/bitset.c: In function ‘mutbitset_iop_PyLongObject’:
      src/sets/bitset.c:1932:9: error: implicit declaration of function ‘_PyLong_Frexp’; did you mean ‘_PyLong_New’? [-Wimplicit-function-declaration]
       1932 |     x = _PyLong_Frexp((PyLongObject *)v, &e);
            |         ^~~~~~~~~~~~~
            |         _PyLong_New
      src/sets/bitset.c:1959:9: error: too few arguments to function ‘_PyLong_AsByteArray’
       1959 |     r = _PyLong_AsByteArray((PyLongObject *)v,
            |         ^~~~~~~~~~~~~~~~~~~
      In file included from /usr/include/python3.13/longobject.h:107,
                       from /usr/include/python3.13/Python.h:81,
                       from src/sets/bitset.c:4:
      /usr/include/python3.13/cpython/longobject.h:111:17: note: declared here
        111 | PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v,
            |                 ^~~~~~~~~~~~~~~~~~~
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for guppy3
  Running setup.py clean for guppy3
Failed to build guppy3
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (guppy3)

Anything more I can/should provide?

Thanks!

@zhuyifei1999
Copy link
Owner

Ah I need to add 3.13 support some time. Let me see if I can get to it this weekend

@zhuyifei1999
Copy link
Owner

Compilation is fixed in the 3.13 branch (master...py3.13) but the tests are not passing (yet).

@zhuyifei1999
Copy link
Owner

Tests are passing locally as of 7541c2b

@zhuyifei1999
Copy link
Owner

There's an odd test failure with coverage=true, where:

imported: guppy.heapy.test.test_View
test_cal_hiding (guppy.heapy.test.test_View.FirstCase.test_cal_hiding) ... ok
test_dominos (guppy.heapy.test.test_View.FirstCase.test_dominos) ... ok
test_exports (guppy.heapy.test.test_View.FirstCase.test_exports) ... ok
test_horizon (guppy.heapy.test.test_View.FirstCase.test_horizon) ... aseq: Expected: b =  Partition of a set of 1 object. Total size = 56 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100       56 100        56 100 list
Got actually  : a =  Partition of a set of 2 objects. Total size = 269 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  50      213  79       213  79 bytes
     1      1  50       56  21       269 100 list
> /home/zhuyifei1999/guppy3/guppy/heapy/test/support.py(104)aseq()
-> pdb.set_trace()
(Pdb) up
> /home/zhuyifei1999/guppy3/guppy/heapy/test/test_View.py(71)test_horizon()
-> self.aseq(hn, iso(x))
(Pdb) p hn
Partition of a set of 2 objects. Total size = 269 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  50      213  79       213  79 bytes
     1      1  50       56  21       269 100 list
(Pdb) p hn[0].sp
 0: hpy().Root.i0_modules['guppy.heapy.View'].__dict__['Horizon'].__dict__['news'].__code__.co_code
(Pdb) p hn[0].prod.all
Traceback (most recent call first):
  File "/home/zhuyifei1999/guppy3/guppy/heapy/View.py", line 17
    def news(self):
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/test_View.py", line 70
    hn = h.news()
  File "/usr/lib/python3.13/unittest/case.py", line 606
    if method() is not None:
  File "/usr/lib/python3.13/unittest/case.py", line 651
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.13/unittest/case.py", line 707
    return self.run(*args, **kwds)
  File "/usr/lib/python3.13/unittest/suite.py", line 122
    test(result)
  File "/usr/lib/python3.13/unittest/suite.py", line 84
    return self.run(*args, **kwds)
  File "/usr/lib/python3.13/unittest/runner.py", line 240
    test(result)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/support.py", line 53
    result = runner.run(suite)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/support.py", line 75
    run_suite(suite, testclass)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/test_View.py", line 296
    support.run_unittest(FirstCase, debug)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/test_all.py", line 36
    f(debug=debug)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/Use.py", line 370
    self._parent.test.test_all.test_main(debug)
  File "/home/zhuyifei1999/guppy3/test.py", line 2
    hpy().test(False)
  File "/home/zhuyifei1999/guppy3/venv.py313/lib/python3.13/site-packages/coverage/execfile.py", line 208
    exec(code, main_mod.__dict__)
  File "/home/zhuyifei1999/guppy3/venv.py313/lib/python3.13/site-packages/coverage/cmdline.py", line 858
    runner.run()
  File "/home/zhuyifei1999/guppy3/venv.py313/lib/python3.13/site-packages/coverage/cmdline.py", line 681
    return self.do_run(options, args)
  File "/home/zhuyifei1999/guppy3/venv.py313/lib/python3.13/site-packages/coverage/cmdline.py", line 970
    status = CoverageScript().command_line(argv)
  File "/home/zhuyifei1999/guppy3/venv.py313/bin/coverage", line 8
    sys.exit(main())

Somehow calling a function lazy compiles the function in 3.13 under certain conditions?!

The test asserts that between these three lines:

h = self.View.horizon()
x = []
hn = h.news()

only a list is allocated. Yet sometimes it shows 1 list object and 1 bytes object, where the bytes is the bytecode of the news function. This is extremely odd...

@gothicVI
Copy link
Author

gothicVI commented Jan 2, 2025

I know way too little about the inner workings of python, but the 3.13 change log (https://docs.python.org/3/whatsnew/changelog.html#changelog) mentions lazy several times.

Once specifically with fixing a bug in 3.12 related to frame loading and performance tracing linking to python/cpython#107674. No idea if this is helpful at all.

EDIT: @zhuyifei1999, could it be that due to the coverage=true the code paths might differ due to instrumentation or tracing by the coverage library, potentially triggering the lazy compilation behavior?

You could enforce compilation via h.news.__code__ if I'm not mistaken (after line 68) and you might log .__code__.co_code in the test to check when the bytecode for news is generated and compare to the coverage=false case.

@zhuyifei1999
Copy link
Owner

zhuyifei1999 commented Jan 2, 2025

No worries I just figured it out with gdb.

C Backtrace:

#0  _PyCode_GetCode (co=0x7f1c4ecbf030) at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Objects/codeobject.c:1667
#1  0x00007f1c50a9f2d9 in PyCode_GetCode (co=<optimized out>) at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Objects/codeobject.c:1696
#2  0x00007f1c4fae0056 in CTracer_handle_call (
    frame=Frame 0x7f1c51065bd0, for file /home/zhuyifei1999/guppy3/guppy/heapy/View.py, line 17, in news (self=<Horizon(mod=<Interface(_share=<Share at remote 0x7f1c4dfcf600>, _owner=<Owner(name='', galog={}, salog={}, inters={<Share at remote 0x7f1c4dfcf100>: <Interface(_share=<...>, _owner=<...>, _name='', guppy=<Interface(_share=<Share at remote 0x7f1c4dfcf240>, _owner=<...>, _name='guppy', heapy=<Interface(_share=<Share at remote 0x7f1c4dfcf2e0>, _owner=<...>, _name='guppy.heapy', Part=<Interface(_share=<Share at remote 0x7f1c4dfcf380>, _owner=<...>, _name='guppy.heapy.Part', _hiding_tag_=[]) at remote 0x7f1c4bb04310>, ImpSet=<Interface(_share=<Share at remote 0x7f1c4dfcf7e0>, _owner=<...>, _name='guppy.heapy.ImpSet') at remote 0x7f1c4bb04520>, Use=<_GLUECLAMP_(_share=<Share at remote 0x7f1c4dfcf9c0>, _owner=<...>, _name='guppy.heapy.Use', _hiding_tag_=[...], iso=<IdentitySetFamily(mod=<Interface(_share=<Share at remote 0x7f1c4dfcf1a0>, _owner=<Owner(name='guppy.heapy.UniSet.fam_IdentitySet', galog={}, salog={}, in...(truncated), self=0x7f1c4ed229a0)
    at /project/coverage/ctracer/tracer.c:557
#3  CTracer_trace (self=0x7f1c4ed229a0, 
    frame=Frame 0x7f1c51065bd0, for file /home/zhuyifei1999/guppy3/guppy/heapy/View.py, line 17, in news (self=<Horizon(mod=<Interface(_share=<Share at remote 0x7f1c4dfcf600>, _owner=<Owner(name='', galog={}, salog={}, inters={<Share at remote 0x7f1c4dfcf100>: <Interface(_share=<...>, _owner=<...>, _name='', guppy=<Interface(_share=<Share at remote 0x7f1c4dfcf240>, _owner=<...>, _name='guppy', heapy=<Interface(_share=<Share at remote 0x7f1c4dfcf2e0>, _owner=<...>, _name='guppy.heapy', Part=<Interface(_share=<Share at remote 0x7f1c4dfcf380>, _owner=<...>, _name='guppy.heapy.Part', _hiding_tag_=[]) at remote 0x7f1c4bb04310>, ImpSet=<Interface(_share=<Share at remote 0x7f1c4dfcf7e0>, _owner=<...>, _name='guppy.heapy.ImpSet') at remote 0x7f1c4bb04520>, Use=<_GLUECLAMP_(_share=<Share at remote 0x7f1c4dfcf9c0>, _owner=<...>, _name='guppy.heapy.Use', _hiding_tag_=[...], iso=<IdentitySetFamily(mod=<Interface(_share=<Share at remote 0x7f1c4dfcf1a0>, _owner=<Owner(name='guppy.heapy.UniSet.fam_IdentitySet', galog={}, salog={}, in...(truncated), what=0x0, arg_unused=<optimized out>)
    at /project/coverage/ctracer/tracer.c:844
#4  0x00007f1c50c15a3b in call_trace_func (self=0x7f1c4ed174b0, arg=None) at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Python/legacy_tracing.c:189
#5  0x00007f1c50c0fed0 in _PyObject_VectorcallTstate (kwnames=0x0, tstate=0x7f1c50efec90 <_PyRuntime+283024>, callable=<sys.legacy_event_handler at remote 0x7f1c4ed174b0>, args=0x80, 
    nargsf=0x8000000000000002) at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Include/internal/pycore_call.h:168
#6  call_one_instrument (interp=interp@entry=0x7f1c50ecf450 <_PyRuntime+88400>, tstate=tstate@entry=0x7f1c50efec90 <_PyRuntime+283024>, args=args@entry=0x7ffc9c063d98, 
    nargsf=nargsf@entry=0x8000000000000002, tool=<optimized out>, event=event@entry=0x0) at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Python/instrumentation.c:907
#7  0x00007f1c50c103ec in call_instrumentation_vector (tstate=0x7f1c50efec90 <_PyRuntime+283024>, event=0x0, instr=<optimized out>, nargs=nargs@entry=0x2, 
    args=args@entry=0x7ffc9c063d90, frame=<optimized out>) at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Python/instrumentation.c:1095
#8  0x00007f1c50c11fcc in call_instrumentation_vector (tstate=<optimized out>, event=<optimized out>, frame=<optimized out>, instr=<optimized out>, nargs=0x2, args=0x7ffc9c063d90)
    at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Python/instrumentation.c:1067
#9  _Py_call_instrumentation (tstate=<optimized out>, event=<optimized out>, frame=<optimized out>, instr=<optimized out>)
    at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Python/instrumentation.c:1132
#10 0x00007f1c50bc18eb in _PyEval_EvalFrameDefault (tstate=0x7f1c4ecbf030, frame=0x7f1c4ecbf0fa, throwflag=0x51065bd0)
    at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Python/generated_cases.c.h:3474
#11 0x00007f1c50a9a37f in _PyObject_VectorcallTstate (tstate=0x7f1c50efec90 <_PyRuntime+283024>, callable=<function at remote 0x7f1c4f0602c0>, args=0x7ffc9c063f80, nargsf=0x2, 
    kwnames=0x0) at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Include/internal/pycore_call.h:168
#12 method_vectorcall (method=<optimized out>, args=0x7f1c4df9f178, nargsf=<optimized out>, kwnames=0x0) at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Objects/classobject.c:92
#13 0x00007f1c50bbe49b in _PyEval_EvalFrameDefault (tstate=0x7f1c4ecbf030, frame=0x7f1c4df9f160, throwflag=0x510659d8)
    at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Python/generated_cases.c.h:1355
#14 0x00007f1c50a986fc in _PyObject_VectorcallDictTstate (tstate=0x7f1c50efec90 <_PyRuntime+283024>, callable=<function at remote 0x7f1c4f0604a0>, args=<optimized out>, nargsf=0x2, 
    kwargs=<optimized out>) at /usr/src/debug/dev-lang/python-3.13.1/Python-3.13.1/Objects/call.c:135

So it's this line
https://github.com/nedbat/coveragepy/blob/10d375b45bce2064ab89938da44eb9287a7ac613/coverage/ctracer/tracer.c#L557

What's odd is that this interaction should have at least existed since 3.12:

python/cpython@b399115#diff-6f869eb8beb7cbe4bc6817584b99ad567f88962fa67f7beca25d009dc401234dR1480
nedbat/coveragepy@956f0fd#diff-68a483552b5cf650f43e371e28ef86ba13f3958e93bc1587c3cb7afad0397f24R537

I then tested with 3.12 and it repros if I only run test_View. Probably some sort of difference where it got primed somehow in 3.12 but not 3.13 while running other tests... But tracemalloc takes forever if I want larger stacks saved.

Edit: Now it repros consistently even when I run all tests in 3.12 🙃 . Not sure I wanna figure out what's going on in coveragepy to see why it started reproing... inconsistently even (didn't repro in https://github.com/zhuyifei1999/guppy3/actions/runs/12582853256)

In any case I'll just prime it, as you said.

EDIT: @zhuyifei1999, could it be that due to the coverage=true the code paths might differ due to instrumentation or tracing by the coverage library, potentially triggering the lazy compilation behavior?

Yes indeed.

@zhuyifei1999
Copy link
Owner

Ok the latest master should be good if you want to test around. I need to make sure the code doesn't die horribly when JIT is enabled (the tests will still probably fail like crazy but I need to make sure at least it doesn't crash Python :) ) before I release a new version.

@zhuyifei1999
Copy link
Owner

zhuyifei1999 commented Jan 2, 2025

As for free-threaded support... I don't think it'd happen. The code already has to work with a lot of CPython's implementation details, and having to lock / unlock everything correctly sounds practically impossible without spending time dedicated to maintaining this and keeping up with every change in Python implementation detail.

Besides... I'll have to lock and unlock every object in the process. The overhead sounds astronomical and almost definitely worse then just having a GIL.

Edit: I could reconsider this if cpython starts exposing _PyEval_EnableGILTransient.

@gothicVI
Copy link
Author

gothicVI commented Jan 2, 2025

Do I need to do anything differently when installing?

$ git clone [email protected]:zhuyifei1999/guppy3.git
$ cd guppy3/
$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip install .
Processing /tmp/guppy3
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: guppy3
  Building wheel for guppy3 (pyproject.toml) ... done
  Created wheel for guppy3: filename=guppy3-3.1.4.post1-cp313-cp313-linux_x86_64.whl size=704547 sha256=b8b38adac29747cb59b3b5c4b232d00486ec0e8fa6ba13b2bf8e9acbcb7a964f
  Stored in directory: /tmp/pip-ephem-wheel-cache-x62_ac2g/wheels/8a/15/40/a1336784ef66a531bba40d24881c19cd17a46929745cbf9089
Successfully built guppy3
Installing collected packages: guppy3
Successfully installed guppy3-3.1.4.post1
(.venv) $ python3
Python 3.13.1 (main, Dec  4 2024, 18:05:56) [GCC 14.2.1 20240910] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import guppy3
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    import guppy3
ModuleNotFoundError: No module named 'guppy3'
>>> import guppy
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    import guppy
  File "/tmp/guppy3/guppy/__init__.py", line 18, in <module>
    from guppy import sets as sets
  File "/tmp/guppy3/guppy/sets/__init__.py", line 1, in <module>
    from guppy.sets.setsc import BitSet        # base bitset type
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'guppy.sets.setsc'

@zhuyifei1999
Copy link
Owner

That's not supposed to happen. And I can repro that. Weird. Hold on.

@zhuyifei1999
Copy link
Owner

zhuyifei1999 commented Jan 2, 2025

Oh I see why.

(.venv) zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-4a /tmp/guppy3 $ python -v
[...]
>>> import guppy
# /tmp/guppy3/guppy/__pycache__/__init__.cpython-313.pyc matches /tmp/guppy3/guppy/__init__.py
# code object from '/tmp/guppy3/guppy/__pycache__/__init__.cpython-313.pyc'
# /tmp/guppy3/guppy/__pycache__/_version.cpython-313.pyc matches /tmp/guppy3/guppy/_version.py
# code object from '/tmp/guppy3/guppy/__pycache__/_version.cpython-313.pyc'
import 'guppy._version' # <_frozen_importlib_external.SourceFileLoader object at 0x7fddb031b410>
# /tmp/guppy3/guppy/etc/__pycache__/__init__.cpython-313.pyc matches /tmp/guppy3/guppy/etc/__init__.py
# code object from '/tmp/guppy3/guppy/etc/__pycache__/__init__.cpython-313.pyc'
import 'guppy.etc' # <_frozen_importlib_external.SourceFileLoader object at 0x7fddb031b590>
# /tmp/guppy3/guppy/etc/__pycache__/Glue.cpython-313.pyc matches /tmp/guppy3/guppy/etc/Glue.py
# code object from '/tmp/guppy3/guppy/etc/__pycache__/Glue.cpython-313.pyc'
import 'guppy.etc.Glue' # <_frozen_importlib_external.SourceFileLoader object at 0x7fddb031b4d0>
# /tmp/guppy3/guppy/sets/__pycache__/__init__.cpython-313.pyc matches /tmp/guppy3/guppy/sets/__init__.py
# code object from '/tmp/guppy3/guppy/sets/__pycache__/__init__.cpython-313.pyc'
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    import guppy
[...]

It's importing from the guppy dir in the current working directory instead of what you installed in site-packages.

You can either:

(.venv) zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-4a /tmp/guppy3 $ cd ..
(.venv) zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-4a /tmp $ python
Python 3.13.1 (main, Dec 30 2024, 00:43:15) [GCC 14.2.1 20241221] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import guppy
>>> guppy.hpy().heap()
Partition of a set of 54630 objects. Total size = 7263282 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  17184  31  1399982  19   1399982  19 str
     1   3287   6  1322448  18   2722430  37 types.CodeType
     2    753   1   856368  12   3578798  49 type
     3  10914  20   793232  11   4372030  60 tuple
     4   7199  13   563782   8   4935812  68 bytes
     5   3140   6   502400   7   5438212  75 function
     6    901   2   296048   4   5734260  79 dict (no owner)
     7    615   1   239896   3   5974156  82 dict of type
     8    133   0   188464   3   6162620  85 dict of module
     9     82   0    88880   1   6251500  86 set
<212 more rows. Type e.g. '_.more' to view.>

or:

(.venv) zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-4a /tmp/guppy3 $ pip install -e .
Obtaining file:///tmp/guppy3
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: guppy3
  Building editable for guppy3 (pyproject.toml) ... done
  Created wheel for guppy3: filename=guppy3-3.1.4.post1-0.editable-cp313-cp313-linux_x86_64.whl size=6186 sha256=b5b17bc47bbbb4d196512307204bd9dd1aa0b60807f5ed40831cf857a552c489
  Stored in directory: /tmp/pip-ephem-wheel-cache-26t1p95h/wheels/8a/15/40/a1336784ef66a531bba40d24881c19cd17a46929745cbf9089
Successfully built guppy3
Installing collected packages: guppy3
  Attempting uninstall: guppy3
    Found existing installation: guppy3 3.1.4.post1
    Uninstalling guppy3-3.1.4.post1:
      Successfully uninstalled guppy3-3.1.4.post1
Successfully installed guppy3-3.1.4.post1
(.venv) zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-4a /tmp/guppy3 $ python
Python 3.13.1 (main, Dec 30 2024, 00:43:15) [GCC 14.2.1 20241221] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import guppy
>>> guppy.hpy().heap()
Partition of a set of 57983 objects. Total size = 7714869 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  17503  30  1426115  18   1426115  18 str
     1   3534   6  1420056  18   2846171  37 types.CodeType
     2    773   1   882160  11   3728331  48 type
     3  11613  20   844128  11   4572459  59 tuple
     4   8650  15   753748  10   5326207  69 bytes
     5   3378   6   540480   7   5866687  76 function
     6    959   2   304640   4   6171327  80 dict (no owner)
     7    635   1   250040   3   6421367  83 dict of type
     8    143   0   196064   3   6617431  86 dict of module
     9   1084   2    86720   1   6704151  87 weakref.ReferenceType
<218 more rows. Type e.g. '_.more' to view.>

Or:

zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-4a /tmp/tmp $ python3.13 -m venv .venv
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-4a /tmp/tmp $ source .venv/bin/activate
(.venv) zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-4a /tmp/tmp $ pip install https://github.com/zhuyifei1999/guppy3/archive/refs/heads/master.zip
Collecting https://github.com/zhuyifei1999/guppy3/archive/refs/heads/master.zip
  Downloading https://github.com/zhuyifei1999/guppy3/archive/refs/heads/master.zip
     \ 563.3 kB 4.2 MB/s 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: guppy3
  Building wheel for guppy3 (pyproject.toml) ... done
  Created wheel for guppy3: filename=guppy3-3.1.4.post1-cp313-cp313-linux_x86_64.whl size=359847 sha256=f568cc35eadea7e92f7b5fa600a1aa5ecd91f93db99fd458772ce422da95af97
  Stored in directory: /tmp/pip-ephem-wheel-cache-u_jd310j/wheels/24/3e/6e/460618de109018fa45ab60c7a3e7efe4199f1414bd2fedde29
Successfully built guppy3
Installing collected packages: guppy3
Successfully installed guppy3-3.1.4.post1
(.venv) zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-4a /tmp/tmp $ python
Python 3.13.1 (main, Dec 30 2024, 00:43:15) [GCC 14.2.1 20241221] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import guppy
>>> guppy.hpy().heap()
Partition of a set of 54101 objects. Total size = 7192727 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  16654  31  1362227  19   1362227  19 str
     1   3287   6  1322448  18   2684675  37 types.CodeType
     2    753   1   856368  12   3541043  49 type
     3  10914  20   793232  11   4334275  60 tuple
     4   7199  13   563770   8   4898045  68 bytes
     5   3140   6   502400   7   5400445  75 function
     6    901   2   296048   4   5696493  79 dict (no owner)
     7    615   1   239896   3   5936389  83 dict of type
     8    133   0   188464   3   6124853  85 dict of module
     9   1186   2    85392   1   6210245  86 types.WrapperDescriptorType
<212 more rows. Type e.g. '_.more' to view.>

@gothicVI
Copy link
Author

gothicVI commented Jan 2, 2025

OMG I forgot the -e... Sorry... that's embarrassing...

@gothicVI
Copy link
Author

gothicVI commented Jan 2, 2025

I can confirm that version a0ef779 seems to work fine now for python 3.13 in the way it did for 3.12 for us.

Thanks!

@zhuyifei1999
Copy link
Owner

I need to make sure the code doesn't die horribly when JIT is enabled (the tests will still probably fail like crazy but I need to make sure at least it doesn't crash Python :) ) before I release a new version.

Surprisingly JIT passes all tests without having to modify the code. But it's fun to see things one isn't normally supposed to:

>>> hp.Root.i0_optimizer
<uop_optimizer object at 0x7f34d591bd90>
>>> hp.Root.i0_executor_list_head
<uop_executor object at 0x560bcd22e760>
>>> dir(hp.Root.i0_optimizer)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>> dir(hp.Root.i0_executor_list_head)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__len__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'get_jit_code', 'get_oparg', 'get_opcode', 'is_valid']

@zhuyifei1999
Copy link
Owner

Alright I released 3.1.5 with the latest changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants