[Model] Switch to Fused RMS norm in Qwen2.5_VL model.

[vllmellm]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

This update introduces support for fused RMSNorm in Qwen2.5 vision language model.

Switching to a fused RMS normalization implementation simplifies the computation trace and enables further optimizations for greater performance improvements in the future. These optimizations may include increased computational efficiency and reduced memory usage by eliminating unnecessary intermediate tensors, decreasing global memory traffic, and improving GPU utilization.

Importantly, these changes do not negatively impact model performance in terms of accuracy or speed.

Benchmark results on Qwen/Qwen2.5-VL-7B-Instruct

Metric	before changes	after changes
Successful requests	1000	1000
Benchmark duration (s)	440.27	440.89
Total input tokens	94327	94327
Total generated tokens	114264	114095
Request throughput (req/s)	2.27	2.27
Output token throughput (tok/s)	259.53	258.78
Total token throughput (tok/s)	473.78	472.73
Mean TTFT (ms)	194967.43	195009.28
Median TTFT (ms)	168388.42	164453.54
P99 TTFT (ms)	428668.77	431252.04
Mean TPOT (ms)	2181.98	2196.16
Median TPOT (ms)	2288.50	2300.13
P99 TPOT (ms)	4544.71	4581.73
Mean ITL (ms)	2009.94	2021.27
Median ITL (ms)	63.95	67.49
P99 ITL (ms)	8188.98	5201.31

Test Plan

Use mistral-evals repo for accuracy evaluation.

step1:

python3 -m eval.run eval_vllm --model_name Qwen/Qwen2.5-VL-7B-Instruct --url http://0.0.0.0:8000

step2:

VLLM_ROCM_USE_AITER=1 \ SAFETENSORS_FAST_GPU=1 \ MIOPEN_FIND_MODE=FAST \ vllm serve Qwen/Qwen2.5-VL-7B-Instruct --trust_remote_code -tp 4

Test Results

eval result on Qwen/Qwen2.5-VL-7B-Instruct model

before changes:

{
"explicit_prompt_relaxed_correctness": 0.8644,
"anywhere_in_answer_relaxed_correctness": 0.8644
}

after changes:

{
"explicit_prompt_relaxed_correctness": 0.8644,
"anywhere_in_answer_relaxed_correctness": 0.8644
}

[gemini-code-assist]

Code Review

This pull request introduces a performance optimization in the Qwen2_5_VisionBlock by switching to a fused RMS normalization kernel. The change correctly refactors the forward pass to use the fused fused_add_rms_norm operation, which combines the residual addition and layer normalization steps. This is a good optimization that maintains the original logic while potentially improving performance by reducing kernel launch overhead and memory traffic. The implementation is clean, correct, and well-contained within the vision block.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

LGTM

[tjtanaa]

CC. @wuhuikx