---
title: '60064'
pageLayout: default
slug: thoughts/inference/vllm-prs/60064
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/60064.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#60064](https://github.com/aarnphm/aarnphm.github.io/issues/60064) \[Model\]\[DeepSeek-V4.1\] Permute mega-attention wq_b / wo_a while loading

[vllm-project/vllm#60064](https://github.com/vllm-project/vllm/pull/60064) · aoshen02 · MERGED · created 2026-10-05T14:07:43Z · merged 2026-10-06T05:20:10Z

<!-- markdownlint-disable -->

## Overview

DeepSeek-V4.1’s FlashMLA mega attention (the SM100 default) permutes `wq_b` rows and `wo_a` columns into the kernel’s layout as a post-load step. That step is only correct when it runs before the linear quant hooks. This PR does the permutation in the weight loaders instead, so it no longer depends on post-load order. It is a prerequisite for [#57733](https://github.com/aarnphm/aarnphm.github.io/issues/57733).

## Claims

- Fixes the DSv4.1 dummy-load startup crash on SM100 (DP4: `IndexError: index 1280 ...`; TP4: a size-mismatch `RuntimeError`).
- Fixes `reload_weights` (layerwise reload) for mega attention. On `main` the reloaded `wq_b` / `wo_a` are written in checkpoint order and `_fused_layouts_ready` skips the permutation, so accuracy collapses. Layerwise reload replays the loaders recorded at model init, so with this PR the reloaded shards are permuted again.
- Real loads are unchanged: after a TP4 cold load, checksums of every parameter and buffer (1605 per rank, after the quant hooks) match `main` on all 4 ranks.
- Removes `finalize_loaded_weights`, `_fused_layouts_ready`, `finalize_mega_attn_weights` and DSpark’s `_finalize_attn` (+62 / −94).

## Validation

DeepSeek-V4.1-Flash, real weights, `vllm serve`, GB200, mega attention (`FLASHMLA_MEGA_ATTN_DSV41`, confirmed in the logs). GSM8K: 1319 questions, 5-shot, greedy. MegaMoE unless noted.

| config                                                         | `main`                                                                                              | this PR                     |
| -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | --------------------------- |
| DP4 / EP4, mega attention                                      | 0.892                                                                                               | 0.896                       |
| TP4 / EP4, mega attention                                      | 0.910                                                                                               | 0.911                       |
| TP4 / EP4, default MoE backend: GSM8K, `reload_weights`, GSM8K | 0.897 → **0.000**                                                                                   | 0.906 → **0.898**           |
| DP4 / EP4 + DSpark k=5: GSM8K, draft acceptance                | 0.892, 51%                                                                                          | 0.896, 51%                  |
| DP4 / EP4, `--load-format dummy`                               | startup crash: `IndexError: index 1280 is out of bounds for dimension 0 with size 1280`             | starts, serves a completion |
| TP4 / EP4, `--load-format dummy`                               | startup crash: `RuntimeError: The size of tensor a (327680) must match the size of tensor b (1280)` | starts, serves a completion |

With MegaMoE, `reload_weights` collapses on both arms for a separate reason: the model-level post-load step (MegaMoE and mHC state) is not rerun after the layerwise reload finishes. [#57733](https://github.com/aarnphm/aarnphm.github.io/issues/57733) reruns it; with [#57733](https://github.com/aarnphm/aarnphm.github.io/issues/57733) and this PR, the same TP4 MegaMoE reload measures 0.901 → 0.898, while [#57733](https://github.com/aarnphm/aarnphm.github.io/issues/57733) without this PR measures 0.900 → 0.002. The DP4 reload is not measurable on any arm: layerwise reload re-materializes the CPU-offloaded Engram table on the GPU and runs out of memory (pre-existing).

`tests/kernels/test_dsv41_mega_attn_layouts.py`: 2 passed. It loads `wq_b` / `wo_a` and their MXFP8 scales through the permuting loader and checks the permuted GEMMs against a reference that permutes the activation instead.

`pre-commit` passes.

Setup: the vLLM image `nightly-dev-arm64-cu130-ac9126e58aa7` plus this commit, overlaid file by file with SHA-256 gates.

## Details

**Root cause.** `permute_wq_b_` derives `head_dim` from `weight.shape[0]`. After the CuteDSL linear hook, `wq_b` is transposed to `[K, N]`, so `head_dim` is wrong and `q_fused_permutation` indexes out of bounds. The `wo_a` BMM hook likewise reshapes `wo_a` to 3D and packs its scales. A real load calls the permutation from inside `load_weights`, before the hooks, so it works there. A dummy load never calls `load_weights`. The model-level `process_weights_after_loading` then runs after the hooks and crashes.

**Fix.** `permute_on_load(param, perm, dim)` wraps the parameter’s `weight_loader` with `composed_weight_loader`, so each loaded shard is gathered into the fused layout right after it is written. The wrapping happens in `DeepseekV4MegaAttnAttention.__init__`, for `wq_b.weight`, `wq_b.weight_scale`, `wo_a.weight` and `wo_a.weight_scale`. Each of these tensors is loaded by a single `weight_loader` call (no stacked mapping), so the whole local shard is present when the permutation runs. The quant hooks then see already-permuted tensors, exactly as `main`’s real load does.

The permutations are computed from the module’s own `n_local_heads` / `head_dim` instead of tensor shapes. `n_local_heads // n_local_groups == WV_GROUP_SIZE` is already enforced in `__init__`.

**Not a duplicate.** I searched open PRs for `mega attention wq_b`, `DeepSeek-V4.1 dummy load`, `fused_layout permute`, `finalize_loaded_weights` and `V4.1 mega attn`. [#57977](https://github.com/aarnphm/aarnphm.github.io/issues/57977) touches `flash_mla_mega_attn.py` for MXFP8 activations but does not change the weight permutation.

**AI assistance.** Written with Claude Code. The submitter reviewed every changed line and ran the tests and GPU checks above.

---

<details>
<summary> Pull Request Checklist </summary>

- [x] I used vLLM’s `/pr-checklist` skill. (Mandatory for agents, optional for humans).

- [x] AI assistance was used during the creation of this PR.

- [x] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

- [x] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

- [x] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.

- [x] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**aoshen02** (2026-10-06T04:11:26Z):

/ci run

**github-actions** (2026-10-06T04:11:45Z):

✅ Triggered [Buildkite CI #93026](https://buildkite.com/vllm/ci/builds/93026) for commit `550a6b5079ea`.

**vllm-agent** (2026-10-06T04:24:55Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 21 test steps (56 jobs) instead of 33 (46 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 33 (46)       | 21 (56)  | 24 (29)    | 12 (39)   |
| AMD mirrors            | 31 (39)       | 3 (3)    | 29 (37)    | 1 (1)     |

<details><summary>Selector would run (21)</summary>

- `arm-cpu-test` ×3
- `basic-models-tests-extra-initialization` ×14
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `cpu-kernel-tests` ×2
- `cpu-language-generation-and-pooling-model-tests` ×3
- `cpu-multi-modal-model-tests-n` ×4
- `deepseek-v4-kernel-test-b200`
- `deepseek-v4-kernel-test-h100`
- `distributed-dp-tests-4-gpus`
- `distributed-model-tests-2-gpus` ×3
- `engine`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-extra-standard` ×2
- `language-models-tests-standard`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `quantization` ×4
- `v1-attention-b200` ×2
- `v1-attention-h100-mi300` ×2

</details>

<details><summary>Would skip (today's rules run them) (24)</summary>

- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `benchmarks-cli-test`
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`

</details>

<details><summary>Would add (today's rules do not run them) (12)</summary>

- `arm-cpu-test` ×3 (code map)
- `basic-models-tests-extra-initialization` ×14 (code map)
- `cpu-kernel-tests` ×2 (code map)
- `cpu-multi-modal-model-tests-n` ×4 (code map)
- `deepseek-v4-kernel-test-b200` (code map)
- `deepseek-v4-kernel-test-h100` (code map)
- `distributed-model-tests-2-gpus` ×3 (code map)
- `engine` (code map)
- `language-models-tests-extra-standard` ×2 (code map)
- `quantization` ×4 (code map)
- `v1-attention-b200` ×2 (code map)
- `v1-attention-h100-mi300` ×2 (code map)

</details>

<details><summary>AMD mirrors: would skip (29)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `distributed-dp-tests-4-gpus`
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`

</details>

<details><summary>AMD mirrors: would add (1)</summary>

- `deepseek-v4-kernel-test-h100` (code map)

</details>

<sub>5 changed files · base `a3e0243b1c` · head `550a6b5079` · Python record: build 92956 at `18c8a65edd` · kernel record: table [`18c8a65`](https://github.com/aarnphm/aarnphm.github.io/commit/18c8a65edd) (build 92956), map [`18c8a65`](https://github.com/aarnphm/aarnphm.github.io/commit/18c8a65edd) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 4 optional steps the selector would also run</sub>

