Skip to content

[Frontend] run-batch supports V1 #21541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 25, 2025
Merged

[Frontend] run-batch supports V1 #21541

merged 1 commit into from
Jul 25, 2025

Conversation

DarkLight1337
Copy link
Member

@DarkLight1337 DarkLight1337 commented Jul 24, 2025 •
edited by github-actions bot
Loading

Essential Elements of an Effective PR Description Checklist

  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Refactor run_batch.py to support V1 by using build_async_engine_client from api_server.py

Test Plan

Updated the test for run-batch to run on V1 as well.

Test Result

(Optional) Documentation Update

@DarkLight1337 DarkLight1337 requested review from mgoin and njhill July 24, 2025 16:53
@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 24, 2025
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added frontend performance Performance-related issues labels Jul 24, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 24, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors run_batch.py to support the V1 engine by reusing the build_async_engine_client function from the API server. The changes are well-structured and enable V1 support as intended. I've identified one potential issue regarding a change in behavior for V0 runs that could impact performance, and I've provided a suggestion to address it.

vllm/entrypoints/openai/run_batch.py
async with build_async_engine_client(
args,
usage_context=UsageContext.OPENAI_BATCH_RUNNER,
disable_frontend_multiprocessing=False,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This change introduces multiprocessing for V0 batch runs by setting disable_frontend_multiprocessing=False. The previous implementation ran the engine in the same process.

For a batch script like run-batch, using a separate process for the engine might introduce unnecessary overhead from inter-process communication and data serialization/deserialization, potentially degrading performance. The in-process engine is generally more suitable for batch jobs.

To maintain the previous behavior for V0 and likely improve performance for batch jobs, I suggest setting this to True. This will not affect V1 runs, as this flag is ignored for the V1 engine.

Suggested change
disable_frontend_multiprocessing=False,
disable_frontend_multiprocessing=True,

DarkLight1337
DarkLight1337 commented Jul 24, 2025
View reviewed changes
vllm/entrypoints/openai/api_server.py
Comment on lines +163 to +164
disable_frontend_multiprocessing = bool(
args.disable_frontend_multiprocessing)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to work around run-batch not having a CLI argument for disable_frontend_multiprocessing. Should we support this for run-batch?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is not much point to this since we're deprecating V0 anyways and this argument is only applicable to V0

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) July 24, 2025 17:07
@vllm-bot vllm-bot merged commit 34ddcf9 into vllm-project:main Jul 25, 2025
70 of 72 checks passed
@DarkLight1337 DarkLight1337 deleted the run-batch-v1 branch July 25, 2025 03:06
liuyumoye pushed a commit to liuyumoye/vllm that referenced this pull request Jul 31, 2025
@DarkLight1337
[Frontend] run-batch supports V1 (vllm-project#21541)
ff6d169 
Signed-off-by: DarkLight1337 <[email protected]>
x22x22 pushed a commit to x22x22/vllm that referenced this pull request Aug 5, 2025
@DarkLight1337 @x22x22
[Frontend] run-batch supports V1 (vllm-project#21541)
14c7ae7 
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025
@DarkLight1337
[Frontend] run-batch supports V1 (vllm-project#21541)
2272b87 
Signed-off-by: DarkLight1337 <[email protected]>
npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025
@DarkLight1337 @npanpaliya
[Frontend] run-batch supports V1 (vllm-project#21541)
4150446 
Signed-off-by: DarkLight1337 <[email protected]>
jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025
@DarkLight1337 @jinzhen-lin
[Frontend] run-batch supports V1 (vllm-project#21541)
81d0237 
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: Jinzhen Lin <[email protected]>
paulpak58 pushed a commit to paulpak58/vllm that referenced this pull request Aug 13, 2025
@DarkLight1337 @paulpak58
[Frontend] run-batch supports V1 (vllm-project#21541)
baec1a1 
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: Paul Pak <[email protected]>
diegocastanibm pushed a commit to diegocastanibm/vllm that referenced this pull request Aug 15, 2025
@DarkLight1337 @diegocastanibm
[Frontend] run-batch supports V1 (vllm-project#21541)
529c188 
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: Diego-Castan <[email protected]>
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025
@DarkLight1337 @epwalsh
[Frontend] run-batch supports V1 (vllm-project#21541)
d3137fa 
Signed-off-by: DarkLight1337 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simon-mo simon-mo simon-mo approved these changes

@mgoin mgoin Awaiting requested review from mgoin

@njhill njhill Awaiting requested review from njhill

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@aarnphm aarnphm Awaiting requested review from aarnphm aarnphm is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
frontend performance Performance-related issues ready ONLY add when PR is ready to merge/full CI is needed v1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@DarkLight1337 @simon-mo @vllm-bot