-
-
Notifications
You must be signed in to change notification settings - Fork 10.4k
[Misc] Fix warnings for mistral model #23552
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: zjy0516 <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request effectively addresses two warnings related to the Mistral model. The first change correctly modifies VoxtralProcessorAdapter
to return a BatchFeature
object, aligning with the expected behavior of ProcessorMixin
. The second set of changes addresses a deprecation warning by explicitly passing the special_token_policy
to the underlying tokenizer methods, which is a good practice for future compatibility. The implementation looks solid and I don't see any critical or high-severity issues.
cc @patrickvonplaten can you check this? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice refactor overall - think we need to address some minor parts but looking good otherwise
Co-authored-by: Patrick von Platen <[email protected]> Signed-off-by: Jiangyun Zhu <[email protected]>
Thanks for your suggestion |
Signed-off-by: zjy0516 <[email protected]>
Signed-off-by: zjy0516 <[email protected]>
I think it' ready to merge. @DarkLight1337 |
Signed-off-by: zjy0516 <[email protected]> Signed-off-by: Jiangyun Zhu <[email protected]> Co-authored-by: Patrick von Platen <[email protected]>
Signed-off-by: zjy0516 <[email protected]> Signed-off-by: Jiangyun Zhu <[email protected]> Co-authored-by: Patrick von Platen <[email protected]>
Purpose
VoxtralProcessorAdapter
returnBatchFeature
pixtral: ditto
pass a special token policy explicitly for self._special_token_policy is deprecated.
Test Plan
Test Result
(Optional) Documentation Update
Essential Elements of an Effective PR Description Checklist
supported_models.md
andexamples
for a new model.