This repository was archived by the owner on Mar 19, 2024. It is now read-only.
Remove the need to iterate the model parameters for freezing and speedup #176
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Differential Revision: D26236748 fbshipit-source-id: 58d615c641af0c042356527a266cf88f4ec25e3c
Differential Revision: D26276800 fbshipit-source-id: 8e6e7ef8447329378983fcc43a840f2b16407bf2
Summary: in swav, there is a parameter that's frozen for 313 iterations. the previous logic iterated through all model named_parameters and checked if the parameter is frozen + iterations <313. This is inefficient as the model grows bigger in size and iterating through all parameters for the check will be slow. alternately, now we check whats the max frozen iteration to which a parameter is frozen and we only iterate named_parameters() of the model if the current iteration <= max frozen iteration inspired by blefaudeux Differential Revision: D26276803 fbshipit-source-id: 75b853f528050cf430d72a2ec712ec2854950822
facebook-github-bot
added
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D26276803 |
facebook-github-bot
pushed a commit
that referenced
this pull request
Feb 9, 2021
…dup (#176) Summary: Pull Request resolved: #176 in swav, there is a parameter that's frozen for 313 iterations. the previous logic iterated through all model named_parameters and checked if the parameter is frozen + iterations <313. This is inefficient as the model grows bigger in size and iterating through all parameters for the check will be slow. alternately, now we check whats the max frozen iteration to which a parameter is frozen and we only iterate named_parameters() of the model if the current iteration <= max frozen iteration inspired by blefaudeux Reviewed By: min-xu-ai Differential Revision: D26276803 fbshipit-source-id: 6d91658273e5ea7fb56ea2c796e0707e90c801b7
facebook-github-bot
pushed a commit
that referenced
this pull request
Aug 6, 2021
Summary: [enhancement] FSDP with activation checkpoint now allows to specify blocks without activations (useful for linear evaluation) and completes incomplete configurations for stage_checkpoints Pull Request resolved: fairinternal/ssl_scaling#176 Reviewed By: prigoyal Differential Revision: D30143386 Pulled By: QuentinDuval fbshipit-source-id: 6fa85059d36d0bfa44ea7c07ac92994985674943
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
in swav, there is a parameter that's frozen for 313 iterations. the previous logic iterated through all model named_parameters and checked if the parameter is frozen + iterations <313. This is inefficient as the model grows bigger in size and iterating through all parameters for the check will be slow.
alternately, now we check whats the max frozen iteration to which a parameter is frozen and we only iterate named_parameters() of the model if the current iteration <= max frozen iteration
inspired by blefaudeux
Differential Revision: D26276803