-
-
Notifications
You must be signed in to change notification settings - Fork 10k
[Models][Quantization] Add quantization configuration update in Voxtral model #24122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Alexandre Marques <[email protected]>
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request adds support for updating quantization configurations in the Voxtral model. The changes introduce a new method to remap module names in the quantization config to match vLLM's internal naming scheme. My review found a couple of critical issues in the implementation of this remapping logic: a duplicated regex pattern that would lead to incorrect mappings, and a faulty condition combined with a missing break
in a loop that would prevent quantization target lists from being updated and could cause multiple transformations on a single name. I've provided suggestions to fix these issues.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Alexandre Marques <[email protected]>
Signed-off-by: Alexandre Marques <[email protected]>
Signed-off-by: Alexandre Marques <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, just a few nits
Co-authored-by: Michael Goin <[email protected]> Signed-off-by: Alexandre Marques <[email protected]>
Signed-off-by: Alexandre Marques <[email protected]>
[Model] This PR updates the quant_config for a Voxtral model (if existent) to map mistralai names to match the vLLM model definition.
This implementation fixes the support of models quantized in the compressed-tensors format being loaded with load_format mistralai.