Add ep #39501
Merged
Add ep #39501
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]>
Merged
|
Does not work for mixtral, tricky because of the sequentiality of the weights. I'll add something that allows for this, but also a plan that allows for one the fly merging the modulelist to the format for megablocks |
25 tasks
|
Will getting deepseek working too be pretty straightforward? |
|
Yes and no! for both I used nn.Modulelist() (deepseek I had no choice and Mixtral I was junior) and so it's a bit more annoying, but yes because next pr will make sur we have a bit of a better interface! |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
commented
Jul 25, 2025
|
[For maintainers] Suggested jobs to run (before merge) run-slow: llama4 |
5 tasks
BenjaminBossan
added a commit
to BenjaminBossan/peft
that referenced
this pull request
Jul 28, 2025
For testing target_parameters, we use a tiny Llama4 model. This model was refactored in huggingface/transformers#39501, resulting in one parameter being accessed an additional time: https://github.com/huggingface/transformers/pull/39501/files#diff-e668ec07f78afdb2cb805d939e47453757f0b9437436cb860fcb7cb2431c9cf5R69 Therefore, a unit test that relied on how often this parameter was accessed started failing. This PR updates the count to the correct number. Additionally debug print statements that were accidentally left over are now removed.
BenjaminBossan
added a commit
to huggingface/peft
that referenced
this pull request
Jul 30, 2025
For testing target_parameters, we use a tiny Llama4 model. This model was refactored in huggingface/transformers#39501, resulting in one parameter being accessed an additional time: https://github.com/huggingface/transformers/pull/39501/files#diff-e668ec07f78afdb2cb805d939e47453757f0b9437436cb860fcb7cb2431c9cf5R69 Therefore, a unit test that relied on how often this parameter was accessed started failing. This PR updates the count to the correct number. Additionally debug print statements that were accidentally left over are now removed.
5 tasks
nvpohanh
added a commit
to nvpohanh/transformers
that referenced
this pull request
Sep 4, 2025
Llama4 accuracy is broken by a bug in huggingface#39501 . It forgot to transpose the router_scores before applying it to routed_in, causing Llama4 to generate garbage output. This PR fixes that issue by adding back the transpose() and adding some comments explaining why the transpose() is needed. Signed-off-by: Po-Han Huang <[email protected]>
Cyrilvallez
added a commit
that referenced
this pull request
Sep 4, 2025
* Fix broken Llama4 accuracy in MoE part Llama4 accuracy is broken by a bug in #39501 . It forgot to transpose the router_scores before applying it to routed_in, causing Llama4 to generate garbage output. This PR fixes that issue by adding back the transpose() and adding some comments explaining why the transpose() is needed. Signed-off-by: Po-Han Huang <[email protected]> * remove comment --------- Signed-off-by: Po-Han Huang <[email protected]> Co-authored-by: Cyril Vallez <[email protected]>
Cyrilvallez
added a commit
that referenced
this pull request
Sep 4, 2025
* Fix broken Llama4 accuracy in MoE part Llama4 accuracy is broken by a bug in #39501 . It forgot to transpose the router_scores before applying it to routed_in, causing Llama4 to generate garbage output. This PR fixes that issue by adding back the transpose() and adding some comments explaining why the transpose() is needed. Signed-off-by: Po-Han Huang <[email protected]> * remove comment --------- Signed-off-by: Po-Han Huang <[email protected]> Co-authored-by: Cyril Vallez <[email protected]>
zaristei
pushed a commit
to zaristei/transformers
that referenced
this pull request
Sep 9, 2025
* EP + updates Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]> * remove unrelated change * not working yet but let's see where it goes! * update the api a bit * udpate * where I am at for now * fix ep * refactor the API * yups * fix * fixup * clean modeling * just support llama4 for now! * properly avoid * fix * nits * Update src/transformers/models/llama4/modeling_llama4.py * Update src/transformers/integrations/tensor_parallel.py * style * ,,,, * update --------- Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]>
zaristei
pushed a commit
to zaristei/transformers
that referenced
this pull request
Sep 9, 2025
* EP + updates Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]> * remove unrelated change * not working yet but let's see where it goes! * update the api a bit * udpate * where I am at for now * fix ep * refactor the API * yups * fix * fixup * clean modeling * just support llama4 for now! * properly avoid * fix * nits * Update src/transformers/models/llama4/modeling_llama4.py * Update src/transformers/integrations/tensor_parallel.py * style * ,,,, * update --------- Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]>
zaristei
pushed a commit
to zaristei/transformers
that referenced
this pull request
Sep 9, 2025
* EP + updates Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]> * remove unrelated change * not working yet but let's see where it goes! * update the api a bit * udpate * where I am at for now * fix ep * refactor the API * yups * fix * fixup * clean modeling * just support llama4 for now! * properly avoid * fix * nits * Update src/transformers/models/llama4/modeling_llama4.py * Update src/transformers/integrations/tensor_parallel.py * style * ,,,, * update --------- Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]>
zaristei
pushed a commit
to zaristei/transformers
that referenced
this pull request
Sep 9, 2025
* EP + updates Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]> * remove unrelated change * not working yet but let's see where it goes! * update the api a bit * udpate * where I am at for now * fix ep * refactor the API * yups * fix * fixup * clean modeling * just support llama4 for now! * properly avoid * fix * nits * Update src/transformers/models/llama4/modeling_llama4.py * Update src/transformers/integrations/tensor_parallel.py * style * ,,,, * update --------- Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]>
zaristei
pushed a commit
to zaristei/transformers
that referenced
this pull request
Sep 9, 2025
* EP + updates Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]> * remove unrelated change * not working yet but let's see where it goes! * update the api a bit * udpate * where I am at for now * fix ep * refactor the API * yups * fix * fixup * clean modeling * just support llama4 for now! * properly avoid * fix * nits * Update src/transformers/models/llama4/modeling_llama4.py * Update src/transformers/integrations/tensor_parallel.py * style * ,,,, * update --------- Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]>
zaristei
pushed a commit
to zaristei/transformers
that referenced
this pull request
Sep 9, 2025
* EP + updates Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]> * remove unrelated change * not working yet but let's see where it goes! * update the api a bit * udpate * where I am at for now * fix ep * refactor the API * yups * fix * fixup * clean modeling * just support llama4 for now! * properly avoid * fix * nits * Update src/transformers/models/llama4/modeling_llama4.py * Update src/transformers/integrations/tensor_parallel.py * style * ,,,, * update --------- Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]>
zaristei
pushed a commit
to zaristei/transformers
that referenced
this pull request
Sep 9, 2025
* EP + updates Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]> * remove unrelated change * not working yet but let's see where it goes! * update the api a bit * udpate * where I am at for now * fix ep * refactor the API * yups * fix * fixup * clean modeling * just support llama4 for now! * properly avoid * fix * nits * Update src/transformers/models/llama4/modeling_llama4.py * Update src/transformers/integrations/tensor_parallel.py * style * ,,,, * update --------- Co-authored-by: Nouamane Tazi <[email protected]> Co-authored-by: drbh <[email protected]>
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Add support for expert parallel!