-
-
Notifications
You must be signed in to change notification settings - Fork 10.4k
[Bugfix] Fix cuda event usage with CPU model runner #23643
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bugfix] Fix cuda event usage with CPU model runner #23643
Conversation
Signed-off-by: jiang1.li <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request introduces a fix for using CUDA events with the CPU model runner by introducing a _torch_cuda_wrapper
to monkey-patch torch.cuda.Event
. The changes look good overall, but there is a critical issue in the implementation of _torch_cuda_wrapper
that could lead to a crash in environments without CUDA support. I've provided a suggestion to make it more robust. The other changes, including using self.pin_memory
and refactoring tensor handling with CpuGpuBuffer
, are solid improvements.
Signed-off-by: jiang1.li <[email protected]> Signed-off-by: tc-mb <[email protected]>
Signed-off-by: jiang1.li <[email protected]>
Signed-off-by: jiang1.li <[email protected]> Signed-off-by: Xiao Yu <[email protected]>
Signed-off-by: jiang1.li <[email protected]>
Signed-off-by: jiang1.li <[email protected]>
Signed-off-by: jiang1.li <[email protected]>
Purpose
_EventPlaceholder
to workaround CUDA event usage withCPUModelRunner
CpuGpuBuffer
.Test Plan
CI tests
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.md
andexamples
for a new model.