Update Python to 3.9 and most other packages as well #1811

akenmorris · 2022-06-30T23:39:32Z

This PR upgrades Python to 3.9 and upgrades almost every package as far as possible.

In particular, we now use Qt from conda, so there is no need to install it anymore.

VTK from conda almost worked, but not quite, so we must still build it. ITK from conda is missing all the headers and was not usable.

happy with our version of the library. Another option is to build VTK as a static library. But I view that as more of a last resort.

The results are slightly different but look roughly equivalent. Some implementation details must have changed in VTK.

Just slightly loosened the tolerance.

…ble with Python 3.9. It crashes at the end: Thread 3063 "python" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ff8a4fa6700 (LWP 164162)] 0x000055555576e7aa in PyThreadState_Clear (tstate=0x7ffa6c000b90) at /usr/local/src/conda/python-3.9.13/Python/pystate.c:788 788 /usr/local/src/conda/python-3.9.13/Python/pystate.c: No such file or directory. Missing separate debuginfos, use: zypper install libglib-2_0-0-debuginfo-2.62.6-3.6.1.x86_64 libglvnd-debuginfo-1.3.2-1.49.x86_64 libgthread-2_0-0-debuginfo-2.62.6-3.6.1.x86_64 (gdb) bt at /usr/local/src/conda/python-3.9.13/Python/pystate.c:788 from /home/csafe/amorris/anaconda3/envs/py39/lib/python3.9/site-packages/torch/lib/libtorch_python.so from /home/csafe/amorris/anaconda3/envs/py39/lib/python3.9/site-packages/torch/lib/libtorch_python.so from /home/csafe/amorris/anaconda3/envs/py39/lib/python3.9/site-packages/torch/lib/libtorch_python.so from /home/csafe/amorris/anaconda3/envs/py39/lib/python3.9/site-packages/torch/lib/libtorch.so Discussion here: pytorch/pytorch#38228 deep_ssm --tiny_test works correclty with PyTorch 1.11.0

Only build VTK with EGL for linux

…den.

Running into this problem on linux container: actions/checkout#760

akenmorris · 2022-07-01T18:56:32Z

Windows failing with this:

c:\bdeps\openvdb\build\openvdb\openvdb\instantiations\gridoperators.cc : fatal error C1002: compiler is out of heap space in pass 2 [C:\bdeps\openvdb\build\openvdb\openvdb\openvdb_shared.vcxproj]

akenmorris · 2022-07-01T18:57:46Z

Linux (both) failing during build_dependencies.sh with this:

The hosted runner: GitHub Actions 6 lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
--

Downgrade libIGL until I can figure out their cmake changes Downgrade boost to 1.74 to be compatible with VTK Install gmock Try building against VTK from conda

GHA is saying: The hosted runner: GitHub Actions 5 lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Apparently conda now has a libattr that /bin/tar from the container doesn't work with.

akenmorris · 2022-07-06T19:04:30Z

Wow, OpenVDB has explicit instantiations that make the libraries >1GB each. This must have been the source of the MSVC compiler out of heap space and was a big contributor to the linux runners running out of disk space. The OpenVDB build dir was >4GB alone.

akenmorris · 2022-07-06T19:24:18Z

Mac OptimizeTests crashing here:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x7240e8f631c7)
  * frame #0: 0x00007240e8f631c7
    frame #1: 0x00007ff8118d0f1e libc++.1.dylib`std::__1::basic_ostream<char, std::__1::char_traits<char> >::operator<<(double) + 238
    frame #2: 0x000000010a48d152 libitkvnl-5.2.1.dylib`vnl_vector_fixed<double, 3u>::print(this=0x00007ff7bfefb718, s=0x00007ff7bfefb508) const at vnl_vector_fixed.hxx:151:14 [opt]
    frame #3: 0x000000010028a3dd OptimizeTests`std::__1::basic_ostream<char, std::__1::char_traits<char> >& operator<<<double, 3u>(ostr=0x00007ff7bfefb508, v=0x00007ff7bfefb718) at vnl_vector_fixed.h:900:5
    frame #4: 0x000000010028a311 OptimizeTests`shapeworks::Constraints::constraintsLagrangianGradient(this=0x0000600003710018, pos=0x00007ff7bfefb768, prepos=0x00007ff7bfefb930, C=1) at Constraints.cpp:509:22
    frame #5: 0x00000001005b4448 OptimizeTests`itk::ParticleGradientDescentPositionOptimizer<float, 3u>::AugmentedLagrangianConstraints(this=0x0000600003008870, gradient=0x00007ff7bfefb8f0, pt=0x00007ff7bfefb930, dom=0x00007ff7bfefb998, maximumUpdateAllowed=0x00007ff7bfefb960) at itkParticleGradientDescentPositionOptimizer.txx:264:29
    frame #6: 0x00000001005b3d77 OptimizeTests`itk::ParticleGradientDescentPositionOptimizer<float, 3u>::StartAdaptiveGaussSeidelOptimization(this=0x0000000119537d58, r=0x0000000119537d40)::'lambda'(tbb::detail::d1::blocked_range<unsigned long> const&)::operator()(tbb::detail::d1::blocked_range<unsigned long> const&) const at itkParticleGradientDescentPositionOptimizer.txx:159:17

Possibly related new warning:

warning: 'operator vnl_vector_ref' is deprecated: Implicit cast conversion is dangerous.

akenmorris · 2022-07-06T19:47:10Z

Windows failing with windeployqt:

2022-07-06T19:37:10.2444315Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: The command "setlocal [C:\build\Studio\src\ShapeWorksStudio.vcxproj]
2022-07-06T19:37:10.2446913Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: C:\Miniconda3\envs\shapeworks\Library\bin\windeployqt.exe --verbose 5 -core -widgets -gui -sql -xml -printsupport --libdir C:/build/bin/Release C:/build/bin/Release/ShapeWorksStudio.exe [C:\build\Studio\src\ShapeWorksStudio.vcxproj]
2022-07-06T19:37:10.2449277Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd [C:\build\Studio\src\ShapeWorksStudio.vcxproj]
2022-07-06T19:37:10.2451138Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: :cmEnd [C:\build\Studio\src\ShapeWorksStudio.vcxproj]
2022-07-06T19:37:10.2452953Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone [C:\build\Studio\src\ShapeWorksStudio.vcxproj]
2022-07-06T19:37:10.2454846Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: :cmErrorLevel [C:\build\Studio\src\ShapeWorksStudio.vcxproj]
2022-07-06T19:37:10.2456611Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: exit /b %1 [C:\build\Studio\src\ShapeWorksStudio.vcxproj]
2022-07-06T19:37:10.2458324Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: :cmDone [C:\build\Studio\src\ShapeWorksStudio.vcxproj]
2022-07-06T19:37:10.2460244Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd [C:\build\Studio\src\ShapeWorksStudio.vcxproj]
2022-07-06T19:37:10.2461974Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: :VCEnd" exited with code 1. [C:\build\Studio\src\ShapeWorksStudio.vcxproj]

Unbelievably, this is apparently just a documented bug with Qt 5.15.4, they somehow managed to publish a release where lconvert can't read their own translation files. Will be fixed in 5.15.5!

For now, it appears we can workaround by disabling translation, which we're not really using anyway.

https://bugreports.qt.io/browse/QTBUG-91558

…ployqt.

https://bugreports.qt.io/browse/QTBUG-91558 Will be fixed in Qt 5.15.5!

akenmorris · 2022-07-09T23:00:53Z

All builds working, binaries working out of the box. Need to double check Studio->Python integration on end user setups.

…n't working anyway, I see no icon on the github docs page. This was causing a problem with the windows build.

akenmorris added 19 commits June 26, 2022 11:28

Upgrading to Python 3.9

18e7aa4

Update ACVD, pyvista

e2bd036

Adding -DVTK_OPENGL_HAS_EGL=True to VTK build so that pyvista will be

efc214e

happy with our version of the library. Another option is to build VTK as a static library. But I view that as more of a last resort.

Use VTK file veresion 42 (backward compatible)

a2826ab

Updating test result output from VTK 9.1.

c9e1cbc

The results are slightly different but look roughly equivalent. Some implementation details must have changed in VTK.

Fix OptimizeTests.procrustes_scale_only_test.

6f9f03b

Just slightly loosened the tolerance.

Update pyvista/vtk API

6ae6aab

Version tag all the updated conda packages

e4dda19

Update pip installed packages

56414b4

Update Mac and Windows GHA to Qt 5.12.12

d27773d

Point to docker image with Qt 5.12.12 for linux

b0167b0

Updating OpenVDB, Eigen and libIGL

db94768

Only build VTK with EGL for linux

Merge branch 'py39' of github.com:SCIInstitute/ShapeWorks into py39

7e56927

Try windows OpenVDB build not in parallel to try to reduce memory bur…

28dbddc

…den.

Update to action/checkout@v3

4cb34c7

Running into this problem on linux container: actions/checkout#760

Still trying to fix git permission issue

cbcf012

Fix EGL flag.

472a06c

Try /Zm250 to try to fix compiler running out of heap space for OpenVDB

3970d43

akenmorris added 9 commits July 1, 2022 13:10

Try MSVC without LTCG to try to fix heap space problem building OpenVDB

36d135f

More build fix attempts

dbdddfc

Trying EGL/GLX options for linux build

ceef186

Bump

b33a5d1

Bump install_shapeworks.sh

141244b

Downgrade itk-elastix to version using current ITK release

acc8fb2

Downgrade libIGL until I can figure out their cmake changes Downgrade boost to 1.74 to be compatible with VTK Install gmock Try building against VTK from conda

Remove shell comment that breaks install

892a18f

Use -DCMAKE_LIBTOOL=/usr/bin/libtool for MacOS

d54a31a

Reduce dep build for linux to 2 procs

477a1ff

GHA is saying: The hosted runner: GitHub Actions 5 lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Set LD_LIBRARY_PATH=/lib/x86_64-linux-gnu for tar

0ba233d

Apparently conda now has a libattr that /bin/tar from the container doesn't work with.

Remove tar testing

e582afa

akenmorris added 12 commits July 6, 2022 15:35

Fix vtk 8.2 to 9.1 in cmake command

bae3b0e

Trying to fix conda python on MacOS

2b25548

Try removing other Qt in case that's causing a problem with the winde…

b2cf3d9

…ployqt.

Linking options for Apple Python with Conda

0e292ae

Try import of vtk after conda install

9ff6c5b

Enable conda for python

3efd362

Set LD_LIBRARY_PATH for linux runner.

3363662

Work around Qt 5.15.4 bug with lconvert via windeployqt

f7036b7

https://bugreports.qt.io/browse/QTBUG-91558 Will be fixed in Qt 5.15.5!

Fiddling with LD_LIBRARY_PATH for linux GHA runner

65094dc

Try importing VTK first

2fdd665

Try adding -DVTK_MODULE_USE_EXTERNAL_VTK_pugixml=1

4df94ac

Add new dlls needed for windows

fdf2a06

Merge branch 'master' into py39

f63666b

akenmorris changed the title ~~Py39 test build~~ Update Python to 3.9 and most other packages as well Aug 11, 2022

akenmorris marked this pull request as ready for review August 11, 2022 23:54

akenmorris requested review from jadie1, iyerkrithika21 and HeavenlyBerserker August 11, 2022 23:55

akenmorris added 4 commits August 12, 2022 11:10

Adding 'pip list' to try to debug why windows mkdocs fails now.

b9fb2b9

Remove fontawesome_markdown. It was only used in one place and it was…

e5cd5ba

…n't working anyway, I see no icon on the github docs page. This was causing a problem with the windows build.

Merge branch 'master' into py39

bffc67e

Fix missing #include

1bdff56

akenmorris merged commit d13f88d into master Aug 14, 2022

akenmorris deleted the py39 branch August 14, 2022 21:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update Python to 3.9 and most other packages as well #1811

Update Python to 3.9 and most other packages as well #1811

Uh oh!

akenmorris commented Jun 30, 2022 •

edited

Loading

Uh oh!

akenmorris commented Jul 1, 2022 •

edited

Loading

Uh oh!

akenmorris commented Jul 1, 2022

Uh oh!

akenmorris commented Jul 6, 2022

Uh oh!

akenmorris commented Jul 6, 2022 •

edited

Loading

Uh oh!

akenmorris commented Jul 6, 2022 •

edited

Loading

Uh oh!

akenmorris commented Jul 9, 2022

Uh oh!

Uh oh!

Update Python to 3.9 and most other packages as well #1811

Update Python to 3.9 and most other packages as well #1811

Uh oh!

Conversation

akenmorris commented Jun 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akenmorris commented Jul 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akenmorris commented Jul 1, 2022

Uh oh!

akenmorris commented Jul 6, 2022

Uh oh!

akenmorris commented Jul 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akenmorris commented Jul 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akenmorris commented Jul 9, 2022

Uh oh!

Uh oh!

akenmorris commented Jun 30, 2022 •

edited

Loading

akenmorris commented Jul 1, 2022 •

edited

Loading

akenmorris commented Jul 6, 2022 •

edited

Loading

akenmorris commented Jul 6, 2022 •

edited

Loading