---
title: '59853'
pageLayout: default
slug: thoughts/inference/vllm-prs/59853
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59853.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59853](https://github.com/aarnphm/aarnphm.github.io/issues/59853) \[Model\] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8

[vllm-project/vllm#59853](https://github.com/vllm-project/vllm/pull/59853) · lesj0610 · MERGED · created 2026-10-03T05:32:09Z · merged 2026-10-10T06:51:53Z

<!-- markdownlint-disable -->

## Overview

Qwen3-VL, Qwen3.5 and Qwen3.8 build their M-RoPE cos/sin cache at 4x `max_position_embeddings`. That headroom exists for Qwen2.5-VL video positions, which these models do not produce, so this sizes the cache like plain RoPE unless video pruning is on.

## Claims

- 4x smaller cos/sin cache per GPU at the default 262k context: 256 → 64 MiB for Qwen3-VL (rotary dim 128), 128 → 32 MiB for Qwen3.5 and Qwen3.8 (rotary dim 64), and 512 → 128 MiB for Qwen3.5 with YaRN at 1M.
- Lower peak while building it, by the same 4x (table below).
- No output change without YaRN, since the new cache is a bit-exact prefix of the old one.
- With YaRN, Qwen3.5’s correction range now uses the original context length, matching Transformers ([#58870](https://github.com/aarnphm/aarnphm.github.io/issues/58870)).

## Validation

Measured on SM80 with each model’s rope parameters:

| model, rope            | cache rows            | cache         | build peak      |
| ---------------------- | --------------------- | ------------- | --------------- |
| Qwen3.5, default 262k  | 1,048,576 → 262,144   | 128 → 32 MiB  | 644 → 161 MiB   |
| Qwen3.5, YaRN x4 1M    | 4,194,304 → 1,048,576 | 512 → 128 MiB | 2,576 → 644 MiB |
| Qwen3-VL, default 262k | 1,048,576 → 262,144   | 256 → 64 MiB  | 1,284 → 321 MiB |

The 262k cache equals the first 262,144 rows of the old one in bf16 and fp32 (`torch.equal`). Against `ROPE_INIT_FUNCTIONS["yarn"]` on a `Qwen3_5TextConfig`, 9 of 32 YaRN inverse frequencies differed before and none differ after.

End to end against main, Qwen3.5-0.8B (`max_model_len=8192`) gives identical prompt logprobs on a 266-token text and identical greedy outputs for text and image prompts without YaRN, and the KV cache gains 15 blocks (60 with YaRN). With YaRN the outputs change, and on that prompt both main and this branch stay at the bf16 noise level against Transformers (mean |Δ logprob| 0.028 and 0.029; 0.025 without YaRN). Qwen3-VL-4B and Qwen3-VL-30B-A3B cut to two layers, on dummy weights in language-model-only mode, give identical greedy tokens and logprobs with the cache at 262,144 rows. With `video_pruning_rate=0.5` the cache keeps 1,048,576 rows. Qwen3.8 Flash-Next was not run end to end; its QSA indexer reads the cache at token positions and at the first position of each compressed group, so both stay in range. `tests/kernels/core/test_mrope.py` and `tests/kernels/core/test_rotary_embedding.py` pass.

<details>
<summary>Cache size check</summary>



```python
from vllm.config import VllmConfig, set_current_vllm_config
from vllm.model_executor.layers.rotary_embedding import get_rope

p = {
  'mrope_interleaved': True,
  'mrope_section': [11, 11, 10],
  'rope_type': 'default',
  'rope_theta': 1e7,
  'partial_rotary_factor': 0.25,
}
with set_current_vllm_config(VllmConfig()):
  for factor in (4, 1):
    rope = get_rope(
      256, 262144, rope_parameters=p, mrope_positions_factor=factor
    )
    print(
      factor, tuple(rope.cos_sin_cache.shape)
    )  # (1048576, 64), (262144, 64)
```

</details>

## Details

The 4x came in with Qwen2.5-VL, where temporal positions scale with video duration and can pass the token count. These models use Qwen3-VL’s positions instead: a frame is a `(1, h, w)` grid spanning `max(h, w)` positions over `h * w` tokens, and text advances by one, so the largest position stays below the token count. Qwen3.8-27B runs as `Qwen3_5ForConditionalGeneration`; Qwen3.8 Flash-Next inherits the same positions.

`MRotaryEmbedding` takes the headroom as `positions_factor`, which defaults to 4. A factor of 1 is passed where each model builds its attention: `Qwen3_5DecoderLayer`, Qwen3-VL’s text model (`Qwen3LLMModel` and `Qwen3VLMoeDecoderLayer`), and Qwen3.8’s QSA attention. Other users of the shared Qwen3, Qwen3-MoE and Qwen3-Next attention keep the 4x, including Qwen3-Omni, whose positions scale with video seconds.

Video pruning keeps the positions of the tokens it drops, so with pruning on the cache stays at 4x. Drafters follow the target’s pruning setting, because some MTP drafters are built from a draft model config that does not carry it. Since pruning now changes the cache shape, `ModelConfig.compute_hash` includes whether it is enabled.

[#58879](https://github.com/aarnphm/aarnphm.github.io/issues/58879) fixes the YaRN correction range for every M-RoPE model and leaves the cache size alone. The two overlap only on Qwen3.5 with YaRN and video pruning disabled, where they give the same frequencies.

AI assistance: Codex, Claude

---

<details>
<summary> Pull Request Checklist </summary>

- [x] I used vLLM’s `/pr-checklist` skill. (Mandatory for agents, optional for humans).

- [x] AI assistance was used during the creation of this PR.

- [x] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

- [x] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

- [x] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.

- [x] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

</details>

**BEFORE SUBMITTING, PLEASE READ <https://docs.vllm.ai/en/latest/contributing>** (anything written below this line will be removed by GitHub Actions)

---

## comments

**cjackal** (2026-10-03T13:48:23Z):

I think Qwen3VL and Qwen3.8 are bound to the same 4x mrope cache problem. Qwen3VL doubles the mrope dimension so the memory gain applies better there. Do you mind having a look at the other qwen family models as well?

**lesj0610** (2026-10-03T14:27:22Z):

@cjackal Qwen3-VL and Qwen3.8 can take the same flag, and Qwen3-VL would save twice as much since its rotary dim is 128 (256 → 64 MiB at 262k). They don’t go through Qwen3_5DecoderLayer, though, and Qwen3-VL-MoE’s text model is shared with the Qwen3-Omni thinker, whose positions scale with video seconds, so the flag has to be set per model there. I’d like to see how this one lands before extending it to those.

**cjackal** (2026-10-03T15:55:08Z):

> @cjackal Qwen3-VL and Qwen3.8 can take the same flag, and Qwen3-VL would save twice as much since its rotary dim is 128 (256 → 64 MiB at 262k). They don’t go through Qwen3_5DecoderLayer, though, and Qwen3-VL-MoE’s text model is shared with the Qwen3-Omni thinker, whose positions scale with video seconds, so the flag has to be set per model there. I’d like to see how this one lands before extending it to those.

I guess qwen3-omni compatibility part is guaranteed - the model definition for [qwen3-omni thinker](https://github.com/vllm-project/vllm/blob/f03026a548b18ba0a99ca0a632b9343f25a66bc9/vllm/model_executor/models/qwen3_omni_moe_thinker.py#L1081) defines its own `Qwen3MoeLLMModel` and we would set the default `mrope_positions_bounded` kwarg to false. On the other hand, current changes in Qwen3.5 may be unsafe for other models inheriting Qwen3.5 - e.g. InternS2 Mobius’ mtp drafter [overrides the model config](https://github.com/vllm-project/vllm/blob/f03026a548b18ba0a99ca0a632b9343f25a66bc9/vllm/model_executor/models/interns2_mobius.py#L446-L459) which unconditionally ignores EVS option under current PR.

Come to think upon the above cases and ease of future extensibility, it would feel better if the ownership for `max_position_embeddings` is on model, not on layer (so to avoid target-draft disparity in general). Maybe a new class attribute at [`SupportsMRoPE`](https://github.com/vllm-project/vllm/blob/f03026a548b18ba0a99ca0a632b9343f25a66bc9/vllm/model_executor/models/interfaces.py#L1743) which is exposed via `ModelConfig`’s property? Then we can set the flag on model definition as class attribute, [similar to this](https://github.com/vllm-project/vllm/blob/4ac0d0eac25eecc98ff519887f2aa3d659dc75f3/vllm/model_executor/models/qwen3_5.py#L471).

**lesj0610** (2026-10-03T16:29:19Z):

@cjackal Omni does define its own `Qwen3MoeLLMModel`, so I had that wrong: Qwen3-VL-MoE can set the flag in `Qwen3VLMoeDecoderLayer` without touching Omni. The drafter case is real for Intern-S2-Mobius. The regular Qwen3.5 MTP keeps the target’s `model_config`, but Mobius swaps in `draft_model_config`, which carries no pruning setting, so its drafter could take the 1x cache while the target prunes. The layer now reads pruning from the target config. I’d keep the flag at the layer here, since it stays a few lines, and revisit ownership if more families pick it up.

**mergify** (2026-10-05T07:27:33Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lesj0610.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-10-05T12:45:34Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lesj0610.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**lesj0610** (2026-10-07T08:44:54Z):

request review @DarkLight1337 @Isotr0py

**Isotr0py** (2026-10-10T05:10:24Z):

/ci run

**github-actions** (2026-10-10T05:10:43Z):

✅ Triggered [Buildkite CI #93983](https://buildkite.com/vllm/ci/builds/93983) for commit `9c3835ffbd9e`.

