[Core] Store only the keys for multi-modal data in P0 #22198
Conversation
Signed-off-by: DarkLight1337 <[email protected]>
DarkLight1337
requested review from
hmellor,
ywang96,
WoosukKwon,
robertgshaw2-redhat,
njhill,
comaniac,
alexm-redhat,
simon-mo,
youkaichao,
mgoin,
tlrmchlsmth,
houseroad and
aarnphm
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
mergify
bot
added
documentation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request effectively separates the multi-modal processor cache from the IPC cache, which is a great architectural improvement for clarity and functionality. The changes across the documentation, configuration, and core engine logic are consistent with this goal. However, I've identified a critical issue in kv_cache_utils.py that could lead to a crash under certain configurations. Please see my detailed comment below.
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
DarkLight1337
changed the title
|
|
||
| ### Adjust CUDA Graph Compilation | ||
| !!! note | ||
| API server scale-out is only available for online inference. |
There was a problem hiding this comment.
Hmmm. What's preventing offline inference from using this?
There was a problem hiding this comment.
That's just how API server scale-out is set up now. Perhaps @njhill can help answer this
| ) | ||
| ``` | ||
| !!! note | ||
| [Multi-modal processor cache](#processor-cache) is disabled when API server scale-out is enabled |
There was a problem hiding this comment.
Link the data_parallel_external_lb doc here as well?
There was a problem hiding this comment.
I think it's not needed because the link is already at the beginning of this section
|
|
||
| ### Multimodal Models | ||
| You can adjust the size of the cache via `VLLM_MM_INPUT_CACHE_GIB` environment variable | ||
| (default 4 GiB per API process + 4 GiB per engine core process). |
| # Cache size (in GiB) for multimodal input cache | ||
| # Default is 4 GiB | ||
| # Cache size (in GiB per process) for multimodal input cache | ||
| # Default is 4 GiB per API process + 4 GiB per engine core process |
Signed-off-by: DarkLight1337 <[email protected]>
github-project-automation
bot
moved this from In Progress
to Done
in Multi-modality Core
…2198) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]>
…2198) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Noam Gat <[email protected]>
…2198) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Paul Pak <[email protected]>
…2198) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Diego-Castan <[email protected]>
…2198) Signed-off-by: DarkLight1337 <[email protected]>
…2198) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Xiao Yu <[email protected]>
…2198) Signed-off-by: DarkLight1337 <[email protected]>
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.Purpose
As discussed with @ywang96, there is actually no need to keep a mirrored cache between P0 and P1. All P0 needs is a key store that mirrors the key store of P1 (i.e. P0 doesn't have to store the values).
This PR separates processor cache (which is used to avoid calling HF processor) from IPC cache (which is used to avoid IPC transfer of multi-modal data from P0 to P1), and adjust the memory usage of each cache (or disable them) separately. This additional flexibility means that only the IPC cache needs to be disabled when DP/API server scale-out and the proposed #21740 are used.This is part of the process of moving the processor cache from P0 to P1. As summarized by @ywang96:
I have removed references to IPC cache from this PR to avoid confusion during this intermediate stage.
cc @WoosukKwon please review this as well!
User-facing changes
- Introduced a new argumentmm_ipc_cache_gbto control the size of the IPC cache.-VLLM_MM_INPUT_CACHE_GIBnow only controls the processor cache. In a future PR, this will be changed to a config argument calledmm_processing_cache_gb, anddisable_mm_preprocessing_cachewill become deprecated (since it is redundant withmm_processing_cache_gb = 0).Test Plan
Test Result
(Optional) Documentation Update