This repository was archived by the owner on Mar 19, 2024. It is now read-only.
Explore ShardedDDP reduce_buffer_size setting to config #177
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Differential Revision: D26236748 fbshipit-source-id: 521c73bdf4fdab110b8e10a20c83da80e9bf0fa2
Summary: as title, in VISSL, we need to set the `reduce_buffer_size=0` as there are parameters that are not actually being used and `find_used_parameters` is something not handled by shardedDPP. setting buffer size to 0 will all reduce the gradients immediately instead of bucketing them Differential Revision: D26276800 fbshipit-source-id: bd976270f20839a9a3c9b057106a6e91f1c18a34
facebook-github-bot
added
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D26276800 |
facebook-github-bot
pushed a commit
that referenced
this pull request
Feb 9, 2021
Summary: Pull Request resolved: #177 as title, in VISSL, we need to set the `reduce_buffer_size=0` as there are parameters that are not actually being used and `find_used_parameters` is something not handled by shardedDPP. setting buffer size to 0 will all reduce the gradients immediately instead of bucketing them Reviewed By: min-xu-ai Differential Revision: D26276800 fbshipit-source-id: 4bbe5a6e3a2b36b8a55abb6e120368025356db17
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: as title, in VISSL, we need to set the
reduce_buffer_size=0as there are parameters that are not actually being used andfind_used_parametersis something not handled by shardedDPP. setting buffer size to 0 will all reduce the gradients immediately instead of bucketing themDifferential Revision: D26276800