Skip to content

Use lazy imports in topology to speed up library import. #942

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: dev
Choose a base branch
from

Conversation

jsmnbom
Copy link

@jsmnbom jsmnbom commented Mar 18, 2025

Loads IPython and scipy only when actually needed. Made build123d load about 1 second (on average) faster on my machine. Python caches modules so all in all this should only be a net benefit.

This is nice when needing to restart the jupyter kernel and having to not wait as long for the library to import.
It would be even better if we could perhaps split the exporters and importers into seperate submodules, and require the user to be explicit about importing them. This would allow ezdxf and svgpathtools to be lazily imported, allowing for about half a second speed up (theoretically).

jsmnbom added 2 commits March 18, 2025 10:31
Loads IPython and scipy only when actually needed. Made build123d load about 1 second faster on my machine. Python caches modules so all in all this should only be a net benefit.
@gumyr
Copy link
Owner

gumyr commented Mar 18, 2025

I'm not convinced about this one. Users can skip reloading all of the build123d (and ocp_vscode) content by following the best practices shown here: https://github.com/bernhard-42/vscode-ocp-cad-viewer?tab=readme-ov-file#best-practices (yes, this should be in the build123d documentation too).

Use the Jupyter extension for a more interactive experience. This allows to have one cell (separated by # %%) at the beginning to import all libraries

# %%
from build123d import *
from ocp_vscode import *

# %%
b = Box(1,2,3)
show(b)
# %%

In addition, there is already a proposal to optimize the IPython content see: #924

@jsmnbom
Copy link
Author

jsmnbom commented Mar 18, 2025

My use case is to have helper code in a separate python package. This helper code needs to import build123d objects.

I thought the only way to refresh the help code in jupyter was to restart the kernel, since IPython extension autoload didn't seem to work - after adddional testing, autoreload does seem to work, but it just behaves weird in quite a few circumstances. All in all I still believe that it makes sense to try to make the library load faster and lazy load heavier dependencies when they are required, but if you do not agree, feel free to close the PR :)

@gumyr
Copy link
Owner

gumyr commented Mar 18, 2025

Getting faster loading times is clearly a good thing. Let me think about it a bit...

@jdegenstein
Copy link
Collaborator

I haven't had time to fully digest, but PEP about lazy imports is here https://peps.python.org/pep-0690/

I personally prefer that lazy imports are not adopted at this time, but I would like to see some more performance profiling on different platforms to understand the impact this could have. Perhaps the profiling could be done on CI for the same platforms we run tests against?

Copy link

codecov bot commented Mar 25, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.93%. Comparing base (518d773) to head (cda904e).
Report is 19 commits behind head on dev.

Additional details and impacted files
@@            Coverage Diff             @@
##              dev     #942      +/-   ##
==========================================
- Coverage   96.93%   96.93%   -0.01%     
==========================================
  Files          32       32              
  Lines        9470     9469       -1     
==========================================
- Hits         9180     9179       -1     
  Misses        290      290              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jsmnbom
Copy link
Author

jsmnbom commented Apr 1, 2025

Do note that that PEP does not actually apply to what this PR does, and was eventually rejected. It only specifies a mechanism for handling how libraries can implement lazy submodules. Whereas all this code change does, is delay the import of a module until it is actually needed.
In terms of how much this would speed up imports I did a quick test:
Before removing scipy as dependency (incl svgpathtools which i missed in this PR)

> uv run python -m timeit -n 1 -r 1 "import build123d" 
1 loop, best of 1: 1.14 sec per loop

after:

>uv run python -m timeit -n 1 -r 1 "import build123d" 
1 loop, best of 1: 832 msec per loop

After reading that PEP I do think build123d should adopt some of what it describes. I see quite a lot of benefit in splitting export and import of different filetypes into submodules. This could even allow users to maybe install the library without most of the dependencies it currently has. Much in a similar way to how VTK is currently somewhat optional (if you can figure out how to compile OCP without).

Supporting lazy imports would also help with #924 and even #946, cutting down the amount of mandatory dependencies to anytree, cadquery-ocp, numpy, trianglesolver, typing-extensions I think.

Other libraries seem to support this using https://peps.python.org/pep-0562/ .

I propose that the exporters and importers should get migrated to a folder structure like this:
image
Where __init__.py uses getattr and dir to only import the other files when necessary.
At the same time I think it would make a lot of sense to refactor some of the exporting and importing code, so that it is more universal across different file types. (this is what the base_*.py files do), and then inside __init__.py can still keep simple user facing and always availible in all and therefore using from build123d import * functions for each format:

def export_step(...):
     from build123d.exporters import ExportStep
    exporter = ExportStep(unit, write_pcurves, precision_mode)
    exporter.add_shape(to_export)
    exporter.write(file_path)

Hope this all makes sense - I got a bit rambly :/

@jsmnbom
Copy link
Author

jsmnbom commented Apr 2, 2025

I'm gonna go ahead and close this PR, as it seems svgpathtools also loads scipy, and that svgpathtools is used a ton inside the exporter for svg. Meaning that it's unfortunately not quite this simple to avoid loading scipy. Instead I will submit a draft PR of my suggestion above soonish :)

Edit: I'm gonna hold off on submitting the draft PR. Started writing it and it is overall looking pretty good in my opinon, but it's also a lot of work, and I don't really see a need to do it, unless y'all think it's a good idea.

@jsmnbom jsmnbom closed this Apr 2, 2025
@jsmnbom
Copy link
Author

jsmnbom commented Apr 2, 2025

Found out how to properly profile this, so I'm gonna open this back up. Sorry for all the back and forth.
Using python -X importime -c "import build123d" and https://kmichel.github.io/python-importtime-graph/ here are two graphs showing 2173ms import time vs. 1010ms import time.

import_time_current
import_time_lazy

I definitely still think there is value in doing this change.

@jsmnbom jsmnbom reopened this Apr 2, 2025
Whoops, forgot pylint warns on non top-level imports
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants