SciPy v1.16.0 #141
SciPy v1.16.0 #141
Conversation
Package Build ResultsTotal packages built: 33 Package Build Times (click to expand)
Longest build: scipy (9m 34s) |
|
Hi @rgommers, I experimented a bit with the f2py generator definition code to make Either way, it's not a blocker here, so this PR is ready for review. Thanks! |
There was a problem hiding this comment.
I am so happy that we are removing this giant patch.
|
Thanks for your super quick reviews on the prerequisite PRs and this PR, @ryanking13! I've rebased the PR to reduce the diff's size, as mentioned. Also, I have changed my mind here, slightly – I'll be working on #144 and adding the full SciPy test suite back, as it should pass on the current SciPy 1.14.1. We will then be able to test any incumbent issues more effectively with this update, rather than relying on a limited set of tests. (That is, we can try and see if it works, at least. I'm still facing the problem of the tests hanging on my macOS machine 😅). |
If it's working now for Pyodide, I'd prefer to not touch this at all right now and keeping the patch, because we're close to proper cross-compilation support in meson-python and then we should be able to get it right at once. |
Can you elaborate? The link says that alternate returns are a Fortran feature with no equivalent in C. |
Makes sense to me, thanks Ralf!
Ah, I should have phrased that better – alternate returns are why we convert |
|
I'll mark this as a draft until we get #148 in. Thanks! |
|
Okay thanks, that is clearer. So it's not that the new C code in SciPy is problematic, but that the OpenBLAS signatures have already been changed and hence the corresponding SciPy code that calls into OpenBLAS needs changing as well. It seems to me that that's a problem inherent in using LAPACK interfaces in Pyodide, and the only way around it is to use LAPACKE instead. Might be worth revisiting at some point after we've finished removing all Fortran code in SciPy. |
Yes, indeed!
I think so too. Until we can explore LAPACKE, LLVM Flang from v20 and up is our best bet as a direct Fortran-to-WASM compiler; pyodide/pyodide#5268 has a few details about previous (partially successful/not yet successful) efforts. |
|
Okay, the full test suite is back now via #148 – I'm going to trigger it and see what new test failures show up. I hope there won't be too many :) |
agriyakhetarpal
added
the
scipy
|
Yes, unfortunately, https://github.com/pyodide/pyodide-recipes/actions/runs/15982819048/job/45082284069?pr=141 has been stuck at this test for the past six minutes (at the time of writing): which I can also replicate locally. We're running the When I encountered this case locally, I skipped this test case; however, many more hangs subsequently appeared and were difficult to debug, as it is not possible to determine ahead of time which tests will produce them. I had eventually given up on running the entire test suite at about 15% completion. |
|
If I run the tests using a Failed to load lib /Users/agriyakhetarpal/Desktop/pyodide-recipes/packages/scipy/.venv-pyodide/lib/python3.13/site-packages/scipy/interpolate/_dierckx.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/pyodide-recipes/packages/scipy/.venv-pyodide/lib/python3.13/site-packages/scipy/interpolate/_dierckx.cpython-313-wasm32-emscripten.so: 0
at Z.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.6.dev3+g5b5bfc7/https_github_com_pyodide_pyodide-build-environment-nightly_releases_download_20250626_xbuildenv_tar_bz2/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:658225)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.6.dev3+g5b5bfc7/https_github_com_pyodide_pyodide-build-environment-nightly_releases_download_20250626_xbuildenv_tar_bz2/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
=================================================================================== test session starts ====================================================================================
platform emscripten -- Python 3.13.2, pytest-8.4.1, pluggy-1.6.0 -- /Users/agriyakhetarpal/Desktop/pyodide-recipes/packages/scipy/.venv-pyodide/bin/python
cachedir: .pytest_cache
hypothesis profile 'default'
rootdir: /Users/agriyakhetarpal/Desktop/pyodide-recipes/packages/scipy
plugins: hypothesis-6.135.20
collected 0 items / 2 errors
========================================================================================== ERRORS ==========================================================================================
___________________________________________________ ERROR collecting .venv-pyodide/lib/python3.13/site-packages/scipy/stats/_bws_test.py ___________________________________________________
ImportError while importing test module '/Users/agriyakhetarpal/Desktop/pyodide-recipes/packages/scipy/.venv-pyodide/lib/python3.13/site-packages/scipy/stats/_bws_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/lib/python313.zip/importlib/__init__.py:88: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv-pyodide/lib/python3.13/site-packages/scipy/stats/__init__.py:626: in <module>
from ._stats_py import *
.venv-pyodide/lib/python3.13/site-packages/scipy/stats/_stats_py.py:52: in <module>
from . import distributions
.venv-pyodide/lib/python3.13/site-packages/scipy/stats/distributions.py:10: in <module>
from . import _continuous_distns
.venv-pyodide/lib/python3.13/site-packages/scipy/stats/_continuous_distns.py:13: in <module>
from scipy.interpolate import BSpline
.venv-pyodide/lib/python3.13/site-packages/scipy/interpolate/__init__.py:192: in <module>
from ._interpolate import *
.venv-pyodide/lib/python3.13/site-packages/scipy/interpolate/_interpolate.py:12: in <module>
from . import _fitpack_py
.venv-pyodide/lib/python3.13/site-packages/scipy/interpolate/_fitpack_py.py:10: in <module>
from ._bsplines import BSpline
.venv-pyodide/lib/python3.13/site-packages/scipy/interpolate/_bsplines.py:10: in <module>
from . import _dierckx
E ImportError: dynamic module does not define module export function (PyInit__dierckx)
_______________________________________________ ERROR collecting .venv-pyodide/lib/python3.13/site-packages/scipy/stats/_page_trend_test.py ________________________________________________
ImportError while importing test module '/Users/agriyakhetarpal/Desktop/pyodide-recipes/packages/scipy/.venv-pyodide/lib/python3.13/site-packages/scipy/stats/_page_trend_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/lib/python313.zip/importlib/__init__.py:88: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv-pyodide/lib/python3.13/site-packages/scipy/stats/__init__.py:626: in <module>
from ._stats_py import *
.venv-pyodide/lib/python3.13/site-packages/scipy/stats/_stats_py.py:52: in <module>
from . import distributions
.venv-pyodide/lib/python3.13/site-packages/scipy/stats/distributions.py:10: in <module>
from . import _continuous_distns
.venv-pyodide/lib/python3.13/site-packages/scipy/stats/_continuous_distns.py:13: in <module>
from scipy.interpolate import BSpline
.venv-pyodide/lib/python3.13/site-packages/scipy/interpolate/__init__.py:192: in <module>
from ._interpolate import *
.venv-pyodide/lib/python3.13/site-packages/scipy/interpolate/_interpolate.py:12: in <module>
from . import _fitpack_py
.venv-pyodide/lib/python3.13/site-packages/scipy/interpolate/_fitpack_py.py:10: in <module>
from ._bsplines import BSpline
.venv-pyodide/lib/python3.13/site-packages/scipy/interpolate/_bsplines.py:10: in <module>
from . import _dierckx
E ImportError: dynamic module does not define module export function (PyInit__dierckx)
================================================================================= short test summary info ==================================================================================
ERROR .venv-pyodide/lib/python3.13/site-packages/scipy/stats/_bws_test.py
ERROR .venv-pyodide/lib/python3.13/site-packages/scipy/stats/_page_trend_test.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================================== 2 errors in 2.80s =====================================================================================Which suggests that I suppose this might also be the reason why I encountered more hangs in the test suite – some more subpackages other than FITPACK may not have been compiled correctly. |
|
All the errors in the test suite are coming from |
suggests that the symbol does exist: and from the And from the So Perhaps I missed patching a |
|
Thanks for the investigation, @agriyakhetarpal. The error happens while loading the library, Because of the error during the loading step, the symbols will not be filled out in the GOT table. That's why the error happens. So I think we'll need to investigate why the library loading fails in the test (It ddidn't fail when I tried to load scipy locally in my browser). To see a more helpful error log, try with a debug build of Pyodide, which is available by passing the following flag |
|
Thanks, @ryanking13! I wonder if we could also release an |
But I think this error only happens in the |
Sounds good to me. Would you like to update https://github.com/pyodide/pyodide-build-environment-nightly to support that feature? |
Oh, actually, another test in TestBSplines (when I was running the test suite locally) was hanging for the same reason (when I skipped this test to allow the test suite to proceed), so I think it must be related to
Yes, I'll put together a PR! |
|
Meanwhile, I ran the hanging test locally and was able to reproduce it in my browser: The |
|
Thanks! Yes, there might be a few more broken libraries besides Perhaps we could test individual test modules ( |
|
With the new debug xbuildenv from https://github.com/pyodide/pyodide-build-environment-nightly/releases/tag/20250703 that we added, I can see this when I try to run a reproducer: Tap to expand logsWhen I check the output from
Update: 20250601 has the same issue, unfortunately. |
|
In the meantime, I pushed a commit to build from 20250703-debug so that we can both have artifacts to examine. |
|
Also, may I ask how you are testing SciPy in your browser? When I copy the Pyodide runtime using the provided script and copy the OpenBLAS libs and other libs from the |
For If you are using the browser, copying the pyodide runtime to the dist directory containing the packages, and running the server in that directory will work (similar to how we do in our CI) |
Description
This PR bumps SciPy to version 1.16.0, released on 22nd June, 2025: https://github.com/scipy/scipy/releases/tag/v1.16.0. Here are a few points:
id_disthas been dropped, and a patch for ARPACK based on the C port ENH:MAINT: sparse.linalg: rewrite ARPACK in C scipy/scipy#22748 has been introduced.linalg.expm,id_dist, and other Fortran codes that are not listed here; please view META: FORTRAN Code inventory scipy/scipy#18566 for more.f2pyentry point directly via Meson generators remains, and we have to resort to usingpython -m numpy.f2pyand custom targets until Stop excluding unisolated packages from build dependencies pyodide-build#21 is resolved.voids are transformed toints in the code for the ARPACK, the_matfuncsfamily, FITPACK, SLSQP, NNLS, and the L-BFGS-B extension modules.id_distpatch,scipy-pytest.js, but the tests always hang on my macOS machine locally at the ~14-15% stage at various points, despite skipping many of them. I suggest that we leave the extra testing for another PR, as the necessary infrastructure to get those tests up and running is a follow-up task that I'd be happy to assign myself to.Tip
This PR depends on https://github.com/pyodide/pyodide-recipes#140, which, at the time of writing, contains a patch that allows compilation to proceed to completion and succeed. That PR also depends on another PR (pyodide/pyodide-build#168).Please review those PRs first (starting with the latter) before coming back to this one, as this will reduce the diff to the relevant files. Thank you!PR stack
Please review the following PRs in the order listed below:
-headerpad_max_install_namesand-dead_strip_dylibsforwasm-ldpyodide-build#168__Pyx_IsSameCFunctionpyodide-build#213pyodide-buildrepository as a submodule #140Additional context
SciPy was previously updated in pyodide/pyodide#5031 to version 1.14.1.
Follow-up tasks after this PR:
numpy-tests(backport of pyodide/pyodide-recipes#86) pyodide#5715unvendor-tests: trueand use-Cinstall-args=--tags=runtime,python-runtime,develTagging the recipe maintainers and our previous collaborators here, like I've done in the past: @lesteve, @steppi, and @rgommers
Thank you for your time!