[Model] Allow the use of sliding window in Qwen2

[inkcherry]

Support the use of sliding window in certain layers of Qwen

FIX #17306
FIX #15705

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[youkaichao]

this comment seems to come from @robertgshaw2-redhat , do you have any ideas?

[heheda12345]

@inkcherry Thanks for the PR! You also need to update the code here to select the proper attention backend when interleaved attention is enabled.

vllm/vllm/config.py

Line 511 in 85b72cb

interleaved_attn_models = ["gemma2", "gemma3_text", "cohere2"]

[heheda12345]

Why do you need to change this file? Can you just pass per_layer_sliding_window=None for not use sliding window?

[inkcherry]

thanks for the review, If per_layer_sliding_window is None, cache_config will be fallback used to determine the value, rather than setting sliding_window to None.

[heheda12345]

I think max_window_layers is only used in qwen series models now. Can you put this function in qwen.py?

[inkcherry]

thanks, moved.

[heheda12345]

@inkcherry Do we really need to support interleaved sliding window? Both QWQ32B and Qwen2.5-7B-Instruct in the issue you linked have the same num_hidden_layers and max_window_layers, meaning that max_window_layers has no effect.

[inkcherry]

@heheda12345 , Yes, this assertion is triggered by the default setting: https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/config.json#L13.
But it's a configurable variable, defined in the model config: https://github.com/huggingface/transformers/blob/716819b8309324302e00a3488a3c3d6faa427f79/src/transformers/models/qwen2/modeling_qwen2.py#L175
This assert crash prevent users from making further modification attempts this feature possibly.

[heheda12345]

I think enabling sliding window is a common need but adjusting max window layer is not.
It needs more discussion about implementing max_window_layers as people may want the following 3 modes and we need to design the API to differentiate them.

1. all layers as full attention
2. first max_window_layers as sliding window and the following layers as full attention
3. all layers as sliding window attention.

What about adding an assertion for num_hidden_layers==max_window_layers when sliding window is enabled and remove the logic for interleaved sliding window? I think it is enough to fix the two issues you mentioned.

[inkcherry]

I think enabling sliding window is a common need but adjusting max window layer is not. It needs more discussion about implementing max_window_layers as people may want the following 3 modes and we need to design the API to differentiate them.

1. all layers as full attention
2. first max_window_layers as sliding window and the following layers as full attention
3. all layers as sliding window attention.

What about adding an assertion for num_hidden_layers==max_window_layers when sliding window is enabled and remove the logic for interleaved sliding window? I think it is enough to fix the two issues you mentioned.

Sure, updated.
By the way, the bottom layers were using SWA, but we ignored this setting.
Currently, only None or full SWA is supported via use_sliding_window

[heheda12345]

LGTM! Thanks for the improvement.

[Yufei-Z]

LGTM! Thanks for the improvement.

I think enabling sliding window is a common need but adjusting max window layer is not. It needs more discussion about implementing max_window_layers as people may want the following 3 modes and we need to design the API to differentiate them.

1. all layers as full attention
2. first max_window_layers as sliding window and the following layers as full attention
3. all layers as sliding window attention.

What about adding an assertion for num_hidden_layers==max_window_layers when sliding window is enabled and remove the logic for interleaved sliding window? I think it is enough to fix the two issues you mentioned.

wait... In transformers' implementation
https://github.com/huggingface/transformers/blob/89c46b648d82b670cc7286a25fa64d2d92770418/src/transformers/models/qwen3/configuration_qwen3.py#L210C9-L216C14

if self.layer_types is None:
self.layer_types = [
"sliding_attention"
if self.sliding_window is not None and i >= self.max_window_layers
else "full_attention"
for i in range(self.num_hidden_layers)
]
num_hidden_layers==max_window_layers means sliding window takes no effect

[heheda12345]

Qwen2 doesn't have sliding window by default. What we want to support is use cache_config.window_size to force running the model with sliding window. non-default max_window_layers is out of scope of this PR.

[vadimkantorov]

Qwen2 doesn't have sliding window by default. What we want to support is use cache_config.window_size to force running the model with sliding window.

Does Qwen3 support sliding window? How could I enable it?

I was surprised that vllm doesn't produce max_tokens output tokens when max_model_len is small, or if prompt is large (and consumes a lot of or even larger than max_model_len tokens)

Thanks :)

[heheda12345]

Qwen3 doesn't have sliding window by default. You can try whether you can force enable it by setting sliding_window. Request will end either the total length reaches max-model-len or the output len reaches max-tokens.

[vadimkantorov]

enable it by setting sliding_window

Somehow pass sliding_window in vllm.LLM(...) args?

Request will end either the total length reaches max-model-len or the output len reaches max-tokens.

For sliding window, is it possible to terminate when the number of response tokens hits some number? Otherwise, it's possible that still long prompt exhausts max-model-len and no response tokens would be generated - which is strange given that this should be technically possible with sliding window

Basically, trying to understand the semantics of sliding_window in vllm