---
title: '59945'
pageLayout: default
slug: thoughts/inference/vllm-prs/59945
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59945.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59945](https://github.com/aarnphm/aarnphm.github.io/issues/59945) \[Bugfix\]\[Qwen4Exp\] Derive the QSA head_dim from the unsharded head count

[vllm-project/vllm#59945](https://github.com/vllm-project/vllm/pull/59945) · lesj0610 · MERGED · created 2026-10-04T12:22:40Z · merged 2026-10-07T12:24:41Z

<!-- markdownlint-disable -->

## Overview

Qwen4Exp QSA computes its fallback `head_dim` from `num_heads`, which is already divided by the tensor-parallel size, so on TP > 1 the fallback is `tp_size` times too large. The fallback is latent: `Qwen4ExpTextConfig` defaults `head_dim` to 256 and rejects `None`, so only an explicit `head_dim: 0` reaches it. This fixes the arithmetic so it matches the `hidden_size // total_num_heads` idiom used elsewhere.

## Claims

- The fallback divides by the unsharded head count, so it is correct on any TP size.
- No change for any config with a nonzero `head_dim`, which covers every real checkpoint.

## Validation



```
pytest tests/models/qwen4_exp/
```

Unchanged against `main` on compute capability 8.0: this is reached only when `config.head_dim` is 0, which no Qwen3.8-Flash-Next config sets, so no existing test exercises the fallback.

The defect is by inspection of the surrounding lines:



```python
self.total_num_heads = int(config.num_attention_heads)
...
self.num_heads = self.total_num_heads // tp_size
...
self.head_dim = int(config.head_dim or self.hidden_size // self.num_heads)
```

`hidden_size` is sized against the unsharded head count, so dividing by the sharded one overstates `head_dim` by exactly `tp_size`. `q_size` and `kv_size` are then built from it, so the QKV projection is created at the wrong width and the checkpoint fails to load with a shape mismatch rather than producing wrong output.

No test added: the fallback has no checkpoint behind it in the suite, and a test would have to fabricate a config with `head_dim: 0` purely to assert one arithmetic expression. The failure mode is a loud shape mismatch, not a silent wrong result.

Not validated end to end: no real checkpoint reaches the fallback.

## Details

The same expression appears on the CUDA and ROCm QSA owners and both are changed. `total_num_heads` is assigned a few lines above the fallback in each, so no reordering is needed.

---

<details>
<summary> Pull Request Checklist </summary>

- [x] I used vLLM’s `/pr-checklist` skill. (Mandatory for agents, optional for humans).

- [x] AI assistance was used during the creation of this PR.

- [x] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

- [x] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

- [x] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.

- [x] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

</details>

**BEFORE SUBMITTING, PLEASE READ <https://docs.vllm.ai/en/latest/contributing>** (anything written below this line will be removed by GitHub Actions)

---

## comments

**lesj0610** (2026-10-07T08:52:16Z):

request review @DarkLight1337 @Isotr0py

**gau-nernst** (2026-10-07T09:12:14Z):

_(Posted by an AI assistant on behalf of @gau-nernst, who reviewed and authorized this comment.)_

LGTM, arithmetic verified: `total_num_heads` is assigned above the fallback in both files, and dividing `hidden_size` by the sharded `num_heads` indeed overstates `head_dim` by exactly `tp_size`. Matches the standard idiom (`hidden_size // total_num_heads`) used elsewhere.

Two notes, neither blocking:

1. **Reachability:** `Qwen4ExpTextConfig` comes from upstream `transformers`, where `head_dim` is a strict field defaulting to `256` — a config JSON that omits it still gets `256`, and `null` is rejected at validation. So this fallback is effectively dead code for any real checkpoint (only an explicit `head_dim: 0` would reach it). The fix is still correct, but the description would be more accurate framed as correcting latent arithmetic rather than an active TP>1 bug.

2. **Same pattern elsewhere:** the parent `Qwen3NextAttention` (`vllm/model_executor/models/qwen3_next.py:300`) and `param2moe.py:113` carry the identical expression. Out of scope here since Qwen4Exp never runs the parent’s `__init__`, but worth a follow-up.

No test is reasonable given the loud shape-mismatch failure mode and no reachable checkpoint.

**gau-nernst** (2026-10-07T09:16:33Z):

/ci run

**github-actions** (2026-10-07T09:16:55Z):

✅ Triggered [Buildkite CI #93273](https://buildkite.com/vllm/ci/builds/93273) for commit `6341a7854cb2`.

**vllm-agent** (2026-10-07T10:05:42Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 0 test steps (0 jobs) instead of 36 (49 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 36 (49)       | 0 (0)    | 36 (49)    | 0 (0)     |
| AMD mirrors            | 33 (41)       | 5 (5)    | 32 (40)    | 4 (4)     |

<details><summary>Selector would run (0)</summary>

none

</details>

<details><summary>Would skip (today's rules run them) (36)</summary>

- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `cpu-language-generation-and-pooling-model-tests` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `qwen4-exp-unit-tests`
- `qwen4-exp-unit-tests-cpu`
- `rl-entrypoints-tests`
- `v1-spec-decode`

</details>

<details><summary>Would add (today's rules do not run them) (0)</summary>

none

</details>

<details><summary>AMD mirrors: would skip (32)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`
- `qwen4-exp-unit-tests`
- `v1-spec-decode`

</details>

<details><summary>AMD mirrors: would add (4)</summary>

- `batch-invariance-h100` (code map)
- `pytorch-nightly-dependency-override-check` (code map)
- `ray-dependency-compatibility-check` (code map)
- `torch-stable-abi-audit` (code map)

</details>

<sub>2 changed files · base `3ca00a8261` · head `17f536603f` · Python record: build 93039 at `1e5d0ea888` · kernel record: table [`385d86a`](https://github.com/aarnphm/aarnphm.github.io/commit/385d86a6cf) (build 93187), map [`385d86a`](https://github.com/aarnphm/aarnphm.github.io/commit/385d86a6cf) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 3 optional steps the selector would also run</sub>

**lesj0610** (2026-10-07T11:00:28Z):

/ci run

**github-actions** (2026-10-07T11:02:01Z):

✅ Triggered [Buildkite CI #93287](https://buildkite.com/vllm/ci/builds/93287) for commit `17f536603f1b`.

