Skip to content

Adds a script to convert NeoX 2.0 checkpoints to DeepSpeed's universal checkpoint format #836

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 115 commits into
base: main
Choose a base branch
from
Open

Adds a script to convert NeoX 2.0 checkpoints to DeepSpeed's universal checkpoint format #836

wants to merge 115 commits into from

Conversation

dashstander
Copy link
Contributor

Modifies the BigScience checkpoint conversion scripts to convert from NeoX DeeperSpeed to either the universal checkpoint format with tools/ds_to_universal.py or to a different DeepSpeed tensor/pipeline parallel format with tools/deepspeed_to_deepspeed.py

When loading a universal checkpoint be sure to add "load_universal": true to your config.

Paired with this DeeperSpeed PR.

dashstander and others added 25 commits January 31, 2023 15:25
@dashstander
Check BigScience scripts
b666034 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Import NeoxCheckpoint
4c1850f 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Different file structure
174a182 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Reformat
e065d44 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Add BigScience universal script
83a4aa6 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Use tokenizer directly
51fa5df 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Use tokenizer directly
ea11b87 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Use tokenizer directly
48c395f 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Use tokenizer directly
1c9aec2 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
build_tokenizer actually returns None
97bfd11 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
If we have the tokenizer we already have the padded vocab size
2ab9ca2 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
If we have the tokenizer we already have the padded vocab size
d2f028e 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Changing universal
7765730 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Changing universal
8dee716 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Need to calculate properly
4f06a6f 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Need to calculate properly
8097e17 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Make our own function
027fb91 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Make our own function
3af69a5 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Make our own function
ad23aca 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Need to reshape with tp in mind
04129bd 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Need to reshape with tp in mind
6b15d57 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Merge branch 'main' into ckpt_reshape
4cd7cfc
@dashstander
Get tokenizer size from tokenizer
cd09987 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Get tokenizer size from tokenizer
04a7829 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Add argument for universal checkpoint
40c4ae9 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander dashstander requested a review from a team as a code owner March 14, 2023 18:59
dashstander and others added 28 commits August 7, 2023 19:25
@dashstander
Make more robust
8d9f324 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
merge
a3f79a3 
Signed-off-by: Dashiell Stander <[email protected]>
Update NeoXArgs docs automatically
0f43323
@dashstander
ok
8fbaa15 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Merge branch 'ckpt_reshape' of https://github.com/EleutherAI/gpt-neox …
54e9a5a 
…into ckpt_reshape
Update NeoXArgs docs automatically
9d39106
@dashstander
ok
5947a66 
Signed-off-by: Dashiell Stander <[email protected]>
Update NeoXArgs docs automatically
00ace51
@dashstander
different printing
58a3c92 
Signed-off-by: Dashiell Stander <[email protected]>
Update NeoXArgs docs automatically
f157814
@dashstander
ok
9903e8e 
Signed-off-by: Dashiell Stander <[email protected]>
Update NeoXArgs docs automatically
8b74404
@dashstander
ok
860161b 
Signed-off-by: Dashiell Stander <[email protected]>
Update NeoXArgs docs automatically
1fec493
@dashstander
blegh
c0682eb 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Merge branch 'ckpt_reshape' of https://github.com/EleutherAI/gpt-neox …
188c379 
…into ckpt_reshape
Update NeoXArgs docs automatically
38b108a
@dashstander
printing
70284f1 
Signed-off-by: Dashiell Stander <[email protected]>
Update NeoXArgs docs automatically
ed59200
@dashstander
change printing
3dea236 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Merge branch 'ckpt_reshape' of https://github.com/EleutherAI/gpt-neox …
206ffea 
…into ckpt_reshape
@dashstander
Get param group info
a697c81 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Get param group info
f51ceee 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Get param group info
9645980 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Get param group info
e9d3000 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Get param group info
c32bc10 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Log bit16 groups
11531ad 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
ok
9ab2cee 
Signed-off-by: Dashiell Stander <[email protected]>
@Quentin-Anthony Quentin-Anthony mentioned this pull request Mar 15, 2024
Closed
@Quentin-Anthony Quentin-Anthony removed the merge-queue This PR is next on the queue to merge label Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Quentin-Anthony Quentin-Anthony Awaiting requested review from Quentin-Anthony Quentin-Anthony was automatically assigned from EleutherAI/pm-gptneo

@StellaAthena StellaAthena Awaiting requested review from StellaAthena StellaAthena was automatically assigned from EleutherAI/pm-gptneo

At least 1 approving review is required to merge this pull request.

Assignees

@Quentin-Anthony Quentin-Anthony

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@dashstander @Quentin-Anthony @StellaAthena @sxthunder @bbo0924 @CLAassistant