Skip to content

Transfer Learning Guide #2394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 21, 2022

Conversation

cgarciae
Copy link
Collaborator

@cgarciae cgarciae commented Aug 10, 2022

What does this PR do?

Adds a Transfer Learning guide which includes:

  • Loading a pre-trained model
  • Doing parameter surgery
  • Freezing layers and implementing Differential Learning Rates using optax.multi_transform

Live preview: https://flax--2394.org.readthedocs.build/en/2394/guides/transfer_learning.html

Notes

  • I couldn't use vit_jax because jax versions conflict, so I used a model from HuggingFace's transformers library. Only quirk here is that transformers downgrades jax so a fix is added to upgrade it on CI.
  • Expanded guide with complete example moved to full transfer learning guide #2429

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@codecov-commenter
Copy link

codecov-commenter commented Aug 10, 2022

Codecov Report

Merging #2394 (2047050) into main (521f516) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #2394      +/-   ##
==========================================
- Coverage   79.47%   79.46%   -0.01%     
==========================================
  Files          49       49              
  Lines        5204     5202       -2     
==========================================
- Hits         4136     4134       -2     
  Misses       1068     1068              
Impacted Files Coverage Δ
flax/linen/stochastic.py 96.42% <0.00%> (-0.24%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@cgarciae cgarciae force-pushed the transfer-learning-guide branch from c256185 to 95f98b3 Compare August 12, 2022 19:54
@cgarciae cgarciae force-pushed the transfer-learning-guide branch from 34a9fb0 to 255cc7a Compare August 29, 2022 21:31
@cgarciae cgarciae added the Priority: P2 - no schedule Best effort response and resolution. We have no plan to work on this at the moment. label Sep 1, 2022
@cgarciae cgarciae force-pushed the transfer-learning-guide branch from 3ee1daa to 58eb73d Compare September 2, 2022 16:26
@cgarciae cgarciae marked this pull request as ready for review September 2, 2022 16:26
@cgarciae cgarciae force-pushed the transfer-learning-guide branch from 21ecc39 to aea323b Compare September 2, 2022 16:57
@cgarciae cgarciae requested a review from jheek September 2, 2022 17:06
@cgarciae cgarciae force-pushed the transfer-learning-guide branch 4 times, most recently from cb98b14 to 2993338 Compare September 21, 2022 16:07
@cgarciae
Copy link
Collaborator Author

Hey @jheek, using jupytext we are syncing the .ipynb and .md files, the .ipynb will be rendered but the .md are there to ease the review process.

@cgarciae cgarciae mentioned this pull request Sep 23, 2022
@8bitmp3 8bitmp3 self-requested a review September 26, 2022 17:22
@cgarciae cgarciae requested a review from jheek September 28, 2022 19:30
@cgarciae cgarciae force-pushed the transfer-learning-guide branch from 8f89edf to 3200c07 Compare September 28, 2022 19:34
@cgarciae
Copy link
Collaborator Author

cgarciae commented Oct 7, 2022

@8bitmp3 thanks a lot for the feedback! It was really useful, I've made the stylistic changes and will keep them in mind for future guides.

@cgarciae cgarciae requested a review from 8bitmp3 October 10, 2022 19:16
@cgarciae cgarciae force-pushed the transfer-learning-guide branch from 359f962 to 76f052f Compare October 17, 2022 15:47
@cgarciae cgarciae requested a review from jheek October 17, 2022 15:52
@cgarciae cgarciae force-pushed the transfer-learning-guide branch from 8250e0b to 6b8016f Compare October 18, 2022 20:47
@8bitmp3
Copy link
Collaborator

8bitmp3 commented Oct 20, 2022

Reviewed and updated in cgarciae#1. LMKWYT!

@cgarciae cgarciae force-pushed the transfer-learning-guide branch from ac2ebde to 2047050 Compare October 20, 2022 15:20
@cgarciae cgarciae force-pushed the transfer-learning-guide branch from c882af3 to cac954e Compare October 20, 2022 20:04
Copy link
Collaborator

@8bitmp3 8bitmp3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @cgarciae

@jheek @andsteing @levskaya @IvyZX If you're happy with our changes, please approve/LGTM. Thanks!

@copybara-service copybara-service bot merged commit fad3965 into google:main Oct 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: P2 - no schedule Best effort response and resolution. We have no plan to work on this at the moment. pull ready
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants