enable loading older TE checkpoints #11930

dimapihtar · 2025-01-22T22:14:10Z

What does this PR do ?

Enables loading older TE checkpoints with avoiding "extra state" layer missmatch error.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: dimapihtar <[email protected]>

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py

Signed-off-by: dimapihtar <[email protected]>

github-actions · 2025-01-24T02:20:55Z

[🤖]: Hi @dimapihtar 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py

Signed-off-by: dimapihtar <[email protected]>

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py

mikolajblaz

just one comment, thanks

Signed-off-by: dimapihtar <[email protected]>

github-actions · 2025-01-24T18:49:55Z

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

Your code was analyzed with PyLint. The following annotations have been identified:

************* Module nemo.collections.nlp.models.language_modeling.megatron_gpt_model
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:334:0: C0301: Line too long (149/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:374:0: C0301: Line too long (136/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:420:0: C0301: Line too long (126/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:467:0: C0301: Line too long (122/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:801:0: C0301: Line too long (131/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1116:0: C0301: Line too long (146/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1137:0: C0301: Line too long (168/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1347:0: C0301: Line too long (122/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1398:0: C0301: Line too long (122/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1455:0: C0301: Line too long (140/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1637:0: C0301: Line too long (132/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1638:0: C0301: Line too long (136/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1640:0: C0301: Line too long (159/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1812:0: C0301: Line too long (128/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1832:0: C0301: Line too long (140/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1840:0: C0301: Line too long (155/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1861:0: C0301: Line too long (141/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1931:0: C0301: Line too long (125/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1958:0: C0301: Line too long (134/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:2007:0: C0301: Line too long (156/119) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:141:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:154:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:180:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:199:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:252:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:290:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:294:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:306:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:314:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:476:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:479:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:711:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:715:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:797:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1114:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1187:12: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1236:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1265:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1468:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1604:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1612:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1621:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:1906:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:2068:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:2075:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:2081:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:20:0: W0611: Unused fields imported from dataclasses (unused-import)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:28:0: W0611: Unused _DataFetcherWrapper imported from lightning.pytorch.loops.fetchers (unused-import)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:30:0: W0611: Unused OmegaConf imported from omegaconf (unused-import)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:72:0: W0611: Unused activation_to_func imported from nemo.collections.nlp.parts.utils_funcs (unused-import)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:81:4: W0611: Unused megatron.core imported as core (unused-import)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:82:4: W0611: Unused tensor_parallel imported from megatron.core (unused-import)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:101:4: W0611: Unused init_method_normal imported from megatron.core.utils (unused-import)
nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py:101:4: W0611: Unused scaled_init_method_normal imported from megatron.core.utils (unused-import)

-----------------------------------
Your code has been rated at 9.49/10

Mitigation guide:

Add sensible and useful docstrings to functions and methods
For trivial methods like getter/setters, consider adding # pylint: disable=C0116 inside the function itself
To disable multiple functions/methods at once, put a # pylint: disable=C0116 before the first and a # pylint: enable=C0116 after the last.

By applying these rules, we reduce the occurance of this message in future.

Thank you for improving NeMo's documentation!

thomasdhc

LGTM

github-actions · 2025-01-24T23:43:49Z

[🤖]: Hi @dimapihtar 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

* set strict to False Signed-off-by: dimapihtar <[email protected]> * add load_legacy flag Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * revert config changes Signed-off-by: dimapihtar <[email protected]> * rename variable Signed-off-by: dimapihtar <[email protected]> * remove extra flag usage Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * change error type Signed-off-by: dimapihtar <[email protected]> * fix warning Signed-off-by: dimapihtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Co-authored-by: dimapihtar <[email protected]> Signed-off-by: Parth Mannan <[email protected]>

* set strict to False Signed-off-by: dimapihtar <[email protected]> * add load_legacy flag Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * revert config changes Signed-off-by: dimapihtar <[email protected]> * rename variable Signed-off-by: dimapihtar <[email protected]> * remove extra flag usage Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * change error type Signed-off-by: dimapihtar <[email protected]> * fix warning Signed-off-by: dimapihtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Co-authored-by: dimapihtar <[email protected]> Signed-off-by: Abhinav Garg <[email protected]>

* set strict to False Signed-off-by: dimapihtar <[email protected]> * add load_legacy flag Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * revert config changes Signed-off-by: dimapihtar <[email protected]> * rename variable Signed-off-by: dimapihtar <[email protected]> * remove extra flag usage Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * change error type Signed-off-by: dimapihtar <[email protected]> * fix warning Signed-off-by: dimapihtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Co-authored-by: dimapihtar <[email protected]> Signed-off-by: Youngeun Kwon <[email protected]>

set strict to False

0b035cc

Signed-off-by: dimapihtar <[email protected]>

github-actions bot added the NLP label Jan 22, 2025

dimapihtar and others added 2 commits January 23, 2025 07:26

add load_legacy flag

5d86690

Signed-off-by: dimapihtar <[email protected]>

Apply isort and black reformatting

5e4dff8

Signed-off-by: dimapihtar <[email protected]>

dimapihtar marked this pull request as ready for review January 23, 2025 15:27

revert config changes

034f43b

Signed-off-by: dimapihtar <[email protected]>

dimapihtar requested a review from mikolajblaz January 23, 2025 15:29

Merge branch 'main' into dpykhtar/fix_ckpt_load

9b95353

janekl reviewed Jan 23, 2025

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Outdated Show resolved Hide resolved

janekl reviewed Jan 23, 2025

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Outdated Show resolved Hide resolved

rename variable

4efd416

Signed-off-by: dimapihtar <[email protected]>

dimapihtar requested a review from janekl January 23, 2025 16:57

dimapihtar and others added 2 commits January 23, 2025 10:50

remove extra flag usage

9cbbc83

Signed-off-by: dimapihtar <[email protected]>

Apply isort and black reformatting

66b9f1b

Signed-off-by: dimapihtar <[email protected]>

dimapihtar added the Run CICD label Jan 23, 2025

janekl reviewed Jan 24, 2025

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Outdated Show resolved Hide resolved

change error type

0d936f0

Signed-off-by: dimapihtar <[email protected]>

dimapihtar added Run CICD and removed Run CICD labels Jan 24, 2025

janekl previously approved these changes Jan 24, 2025

View reviewed changes

mikolajblaz reviewed Jan 24, 2025

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Outdated Show resolved Hide resolved

mikolajblaz approved these changes Jan 24, 2025

View reviewed changes

fix warning

2e48377

Signed-off-by: dimapihtar <[email protected]>

dimapihtar dismissed janekl’s stale review via 2e48377 January 24, 2025 17:15

dimapihtar added Run CICD and removed Run CICD labels Jan 24, 2025

thomasdhc added Run CICD and removed Run CICD labels Jan 24, 2025

thomasdhc approved these changes Jan 24, 2025

View reviewed changes

dimapihtar changed the title ~~set strict to False~~ enable loading older TE checkpoints Jan 24, 2025

dimapihtar enabled auto-merge (squash) January 24, 2025 21:37

dimapihtar merged commit 21e94fc into main Jan 24, 2025
277 of 407 checks passed

dimapihtar deleted the dpykhtar/fix_ckpt_load branch January 24, 2025 23:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

enable loading older TE checkpoints #11930

enable loading older TE checkpoints #11930

Uh oh!

dimapihtar commented Jan 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 24, 2025

Uh oh!

Uh oh!

Uh oh!

mikolajblaz left a comment

Uh oh!

github-actions bot commented Jan 24, 2025

Uh oh!

thomasdhc left a comment

Uh oh!

github-actions bot commented Jan 24, 2025

Uh oh!

Uh oh!

Uh oh!

enable loading older TE checkpoints #11930

enable loading older TE checkpoints #11930

Uh oh!

Conversation

dimapihtar commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 24, 2025

Uh oh!

Uh oh!

Uh oh!

mikolajblaz left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 24, 2025

Uh oh!

thomasdhc left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 24, 2025

Uh oh!

Uh oh!

Uh oh!

dimapihtar commented Jan 22, 2025 •

edited

Loading