Skip to content

prefill metadata splitting #112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 307 commits into
base: sage/dbo-full-cudagraphs
Choose a base branch
from
Draft

prefill metadata splitting #112

wants to merge 307 commits into from
+39,522 −65,238

Conversation

LucasWilkinson
Copy link

@LucasWilkinson LucasWilkinson commented Sep 5, 2025 •
edited by github-actions bot
Loading

Purpose

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

prashantgupta24 and others added 30 commits September 16, 2025 20:58
@prashantgupta24
[misc] fix typo in value error (vllm-project#24995)
ea3de5e 
Signed-off-by: Prashant Gupta <[email protected]>
@russellb
[Core] Get num_encoder_tokens from scheduler config (vllm-project#24989)
58d4c70 
Signed-off-by: Russell Bryant <[email protected]>
@WoosukKwon
[V0 Deprecation] Remove MQLLMEngine (vllm-project#25019)
5801e49 
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
@ywang96 @JJJYmmm
@wulipc @Isotr0py
[Model] Support Qwen3-VL Model Series (vllm-project#24727)
0f7acdd 
Signed-off-by: Roger Wang <[email protected]>
Signed-off-by: Isotr0py <[email protected]>
Co-authored-by: Huang Jie <[email protected]>
Co-authored-by: 松灵 <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
@LucasWilkinson
minor cleanup
aeee2ba 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
more minor cleanup
991a840 
Signed-off-by: Lucas Wilkinson <[email protected]>
@haoyangli-amd @lihaoyang-amd
[Rocm] [quantization] Fix quark ptpc moe and add test case (vllm-proj…
ca2d192 
…ect#24649)

Signed-off-by: Haoyang Li <[email protected]>
Co-authored-by: Haoyang Li <[email protected]>
@pliops-daniels
Add more documentation and improve usability of lognormal dist (bench…
43a62c5 
…mark_serving_multi_turn) (vllm-project#23255)

Signed-off-by: daniels <[email protected]>
@jikunshang
[XPU] Fix xpu model runner call torch.cuda APIs (vllm-project#25011)
dd39baf 
Signed-off-by: Kunshang Ji <[email protected]>
@rouchenzi @abmfy
[EPLB] Support EPLB for Mixtral Model (vllm-project#22842)
b77bf34 
Signed-off-by: rouchenzi <[email protected]>
Signed-off-by: rouchenzi <[email protected]>
Co-authored-by: Bowen Wang <[email protected]>
@lgeiger
[Core][MultiModalHasher] Hash images without converting image mode (v…
03191cd 
…llm-project#24969)

Signed-off-by: Lukas Geiger <[email protected]>
@whx-sjtu
[Model] Pass param prefix to LLMHead (vllm-project#24862)
4a9375f 
Signed-off-by: whx-sjtu <[email protected]>
@whx-sjtu
[Model] Apply SharedFusedMoE to glm4_moe. (vllm-project#24849)
c15309a 
Signed-off-by: whx-sjtu <[email protected]>
@zhuohan123
[Core] Remove tokenizer group in vLLM (vllm-project#24078)
6c47f6b 
Signed-off-by: Zhuohan Li <[email protected]>
@windsonsea
[Docs] Fix griffe warning in base_static_graph.py (vllm-project#25018)
0fb2551 
Signed-off-by: windsonsea <[email protected]>
@xinyu-intel @jikunshang
[DP] Create placement groups by ray_device_key (vllm-project#25026)
bb58dc8 
Signed-off-by: Xinyu Chen <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>
@chaunceyjiang
[Frontend] Support returning all prompt logprobs (vllm-project#24956)
544fe76 
Signed-off-by: chaunceyjiang <[email protected]>
@shijun-yin
[BugFix] enable DOTALL to match multi-line tool_call parameters in ex…
2b85697 
…tract_tool_call_required_streaming (vllm-project#24668)

Signed-off-by: Shijun Yin <[email protected]>
@DarkLight1337
[Misc] Avoid use of deprecated AutoModelForVision2Seq (vllm-project…
e120533 
…#25065)

Signed-off-by: DarkLight1337 <[email protected]>
@danielafrimi
Add RADIO Vision Encoder Support to vLLM (vllm-project#24595)
252ada5 
Signed-off-by: Daniel Afrimi <[email protected]>
Co-authored-by: root <[email protected]>
@bigPYJ1151
[Bugfix] Fix Stream usage in CPU model runner and OneDNN kernel check (…
9fccd04 
…vllm-project#25046)

Signed-off-by: jiang1.li <[email protected]>
@Aidyn-A
Apply fixes for CUDA 13 (vllm-project#24599)
bfe9380 
Signed-off-by: Aidyn-A <[email protected]>
@dolpm @jeejeelee
[fix] lora benchmarks pass no_lora_flag_cpu (vllm-project#23774)
1b962e2 
Signed-off-by: Dylan Maloy <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>
@sighingnow
[Bugfix][Qwen3-Next] fixes the varlen issue in qwen3-next's MTP imple…
dd6a910 
…mentation. (vllm-project#24957)

Signed-off-by: Tao He <[email protected]>
@samzong
[Docs] improve code formatting and comments for eliminate griffe buil…
47f670b 
…d warning. (vllm-project#25010)

Signed-off-by: samzong <[email protected]>
@MatthewBonanni
Remove old cutlass mla (vllm-project#23961)
8f3616f 
Signed-off-by: Matthew Bonanni <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
@samzong
[Docs] vllm/benchmarks/datasets.py fix docstring param format. (vllm-…
4a2d33e 
…project#24970)

Signed-off-by: samzong <[email protected]>
@mgoin
[CI Bugfix] Fix failing test_invalid_env (vllm-project#25078)
087c6ff 
Signed-off-by: mgoin <[email protected]>
@WoosukKwon
[V0 Deprecation] Remove V0 Core tests (vllm-project#25082)
4b946d6 
Signed-off-by: Woosuk Kwon <[email protected]>
@simon-mo @jeejeelee
cleanup: remove adapter commons (vllm-project#25045)
4aa8c7b 
Signed-off-by: Jee Jee Li <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>
DarkLight1337 and others added 30 commits September 22, 2025 11:04
@DarkLight1337
[Misc] Remove unused encoder-decoder error strings (vllm-project#25374)
417a164 
Signed-off-by: DarkLight1337 <[email protected]>
@hmellor
Make pickle import check fast (vllm-project#25379)
64c824c 
Signed-off-by: Harry Mellor <[email protected]>
@hmellor @gemini-code-assist
Make mypy behave like a proper pre-commit hook (vllm-project#25313)
3d2c56b 
Signed-off-by: Harry Mellor <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@Sara-KS
[Kernel] MI-300X triton moe configs (vllm-project#23445)
ac24388 
Signed-off-by: Sara Kokkila Schumacher <[email protected]>
@Csrayz @ivyilike
[Bugfix] Fix several issues with p2p xPyD in GET type (vllm-project#2…
c10101a 
…3993)

Signed-off-by: Csrayz <[email protected]>
Signed-off-by: ivyilike <[email protected]>
Co-authored-by: ivyilike <[email protected]>
@LucasWilkinson @tlrmchlsmth
Update vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finaliz…
99f6c06 
…e.py

Co-authored-by: Tyler Michael Smith <[email protected]>
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
review comments
9416c69 
Signed-off-by: Lucas Wilkinson <[email protected]>
@bringlein
[V1][Attention] Split triton_attn in triton-only and rocm specific ba…
175811e 
…ckends (vllm-project#24648)

Signed-off-by: Burkhard Ringlein <[email protected]>
@abmfy @gemini-code-assist @tlrmchlsmth
[EPLB] Reduce EPLB Inference Overhead (vllm-project#24573)
06a4133 
Signed-off-by: Bowen Wang <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <[email protected]>
@Daisy-Ma-coder @DarkLight1337
[CLI env var] Add VLLM_FLASH_ATTN_MAX_NUM_SPLITS_FOR_CUDA_GRAPH in en…
cfbee3d 
…v variables (vllm-project#25274)

Signed-off-by: qqma <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
Co-authored-by: qqma <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>
@ElizaWszola
[Compiler] Disable Inductor standalone compile by default (vllm-proje…
1d7f95b 
…ct#25391)

Signed-off-by: ElizaWszola <[email protected]>
@mgoin
[CI Failure] Fix fp8 kv cache on <SM90 (vllm-project#25396)
239ef0c 
Signed-off-by: mgoin <[email protected]>
@luccafong @zhuohan123
[DP] support torchrun external launcher with Data Parallelism (vllm-p…
922979b 
…roject#24899)

Signed-off-by: Lu Fang <[email protected]>
Signed-off-by: Zhuohan Li <[email protected]>
Co-authored-by: Zhuohan Li <[email protected]>
@simon-mo
[misc] Remove RFC review hours reference (vllm-project#25416)
8d0ee5a
@ProExpertProg
[torch.compile] Cleanup compilation tests and custom passes, add debu…
d5e0fca 
…g utils, fix DCE bug (vllm-project#23091), fix test (vllm-project#24376), and prep for custom op matching (vllm-project#24604) (vllm-project#24542)

Signed-off-by: Luka Govedič <[email protected]>
Signed-off-by: luka <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
@orozery
[KV offload][5/N] Add CPUOffloadingSpec (vllm-project#24251)
8db2939 
Signed-off-by: Or Ozeri <[email protected]>
@DarkLight1337
[CI/Build] Skip Qwen3-VL initialization tests until models are actual…
f552d5e 
…ly released (vllm-project#25394)

Signed-off-by: DarkLight1337 <[email protected]>
@jcyang43 @yaochengji
[TPU] update torch_xla dependency for PyPI compatibility (vllm-projec…
8bed179 
…t#25278)

Signed-off-by: Johnny Yang <[email protected]>
Co-authored-by: Chengji Yao <[email protected]>
@alecsolder @yeqcharlotte
[Frontend] Responses API MCP tools for built in tools and to pass thr…
45d7d85 
…ough headers (vllm-project#24628)

Signed-off-by: Alec Solder <[email protected]>
Signed-off-by: Alec S <[email protected]>
Co-authored-by: Alec Solder <[email protected]>
Co-authored-by: Ye (Charlotte) Qi <[email protected]>
@ProExpertProg
[Bugfix] fix custom op test (vllm-project#25429)
d588cd2 
Signed-off-by: Luka Govedič <[email protected]>
@russellb
[Core] Drop overly aggressive whisper assertion (vllm-project#25408)
f31ff87 
Signed-off-by: Russell Bryant <[email protected]>
@NickLucche
[Bugfix] Fix missing clear_connector_metadata (vllm-project#25397)
0901970 
Signed-off-by: NickLucche <[email protected]>
@MatthewBonanni @robertgshaw2-redhat @Bam4d
[BugFix] [DP/EP] Fix slow execution when BS <= DP (vllm-project#25407)
ac0048c 
Signed-off-by: Matthew Bonanni <[email protected]>
Co-authored-by: Robert Shaw <[email protected]>
Co-authored-by: Chris Bamford <[email protected]>
@alexm-redhat
[Performance] Remove input pads in cutlass_mla and optimize v_proj ou…
0b7bed9 
…tput handling (vllm-project#25184)

Signed-off-by: Alexander Matveev <[email protected]>
@yewentao256
[Perf] Apply torch.compile for per_block_cast_to_fp8 (vllm-project#…
9949aa2 
…24611)

Signed-off-by: yewentao256 <[email protected]>
@Isotr0py
[V0 deprecation] Remove platform v1 controling interface (vllm-projec…
6fa78d8 
…t#25410)

Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
[V0 deprecation] Remove _set_default_args_v0 function (vllm-project…
c625f90 
…#25409)

Signed-off-by: Isotr0py <[email protected]>
@yewentao256
[Bug] Fix Long Context OOM Issue (vllm-project#25290)
4741239 
Signed-off-by: yewentao256 <[email protected]>
@tlrmchlsmth
Merge branch 'main' into lwilkinson/dbo-prefill
4e9e2ce 
Signed-off-by: Tyler Michael Smith <[email protected]>
@LucasWilkinson
fix pre-commit
356ddcb 
Signed-off-by: Lucas Wilkinson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NickLucche NickLucche Awaiting requested review from NickLucche NickLucche will be requested when the pull request is marked ready for review NickLucche is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin will be requested when the pull request is marked ready for review mgoin is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat will be requested when the pull request is marked ready for review robertgshaw2-redhat is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 will be requested when the pull request is marked ready for review yewentao256 is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth will be requested when the pull request is marked ready for review tlrmchlsmth is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao will be requested when the pull request is marked ready for review youkaichao is a code owner

@njhill njhill Awaiting requested review from njhill njhill will be requested when the pull request is marked ready for review njhill is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat will be requested when the pull request is marked ready for review alexm-redhat is a code owner

@ProExpertProg ProExpertProg Awaiting requested review from ProExpertProg ProExpertProg will be requested when the pull request is marked ready for review ProExpertProg is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.