-
Notifications
You must be signed in to change notification settings - Fork 48
fix: reduce redundant remote_function
deployments
#856
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
def_copy = cloudpickle.loads(cloudpickle.dumps(def_)) | ||
def_copy.__code__ = def_copy.__code__.replace( | ||
co_filename="bigframes_place_holder_filename" | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a more efficient way to do this? Also do we want to replace filename for other code objects in the dependency tree? Seems the most efficient solution would require modifying cloudpickle a bit though
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reg. more efficient way: the most efficient would be to modify one of the pickling routines to take replacements on a live object, I'll consider this in my effort explained down below.
Reg. replacement in the dependency tree: We are not doing dependency tree yet, I'm looking into the cloudpickle (and the native pickle which it builds on top of) to evaluate what modification we can make. Consider this change as an instant work around to reduce re-deployments in two scenarios:
- rerun of a cell in the same session
- rerun of a notebook with explicitly named remote function
See the demo at screencast/cast/NTA0OTk2MTc5MTA5NDc4NHwwYjBjNDBiMS01Mw
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes internal issue #349872122 🦕