---
title: '56935'
pageLayout: default
tags:
  - 56441_
slug: thoughts/inference/vllm-prs/56935
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56935.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56935](https://github.com/aarnphm/aarnphm.github.io/issues/56935) \[Model\]\[DSv4.1\] FlashMLA mega attention and the NVFP4 compressed KV cache

[vllm-project/vllm#56935](https://github.com/vllm-project/vllm/pull/56935) · zyongye · MERGED · created 2026-09-15T03:21:41Z · merged 2026-09-16T10:07:51Z

## Purpose

Supersedes [#56344](https://github.com/aarnphm/aarnphm.github.io/issues/56344), which closed when its base branch `dsv41-optimized` was deleted.

FlashMLA’s mega-attention kernel does Q RoPE, sparse attention, the output’s inverse RoPE and its FP8 cast in one launch, writing straight into the buffer `wo_a` consumes. It arrives as `FlashMLAMegaAttnBackend` (`FLASHMLA_MEGA_ATTN_DSV41`) with its own attention layer, plus the V4.1 NVFP4 compressed KV record that only this kernel can read — and, as of this revision, **it is the SM100 default** rather than an opt-in backend.

Rebased onto current `main`, which now includes [#56441](https://github.com/aarnphm/aarnphm.github.io/issues/56441) (KV-only DSpark context insertion); that PR touched the same translation unit and its interaction with this one is described under \_Rebase onto <ref slug="tags/56441_"> below.

## The interface capability the base layer acts on

`accepts_unnormed_unroped_query` — the attention kernel applies the Q norm and RoPE itself and reads Q in its own chunk-interleaved layout, so the layer’s fused KV insert reduces to a zero-pad of Q to the kernel’s head count, and to nothing at all once the shard is already that wide.

The output side needs no flag: the mega layer’s `_alloc_attn_out` / `_o_proj` pair speaks `QuantizedActivation` where every other layer speaks bf16 and rotates in `_o_proj`. An earlier revision of this PR declared a second capability, `produces_inv_roped_quantized_output`; it had no reader — the dispatch is the override pair — so it is gone.

## Q reaches the kernel through the existing fused op, not a new one

Rather than a separate Triton pad kernel, `fused_deepseek_v4_qnorm_rope_kv_rope_quant_insert` gains two independent knobs:

- `apply_q_rope` — rotate Q as well; KV is always rotated. A runtime argument, on the `kv_mxfp8` precedent.
- `is_q_interleaved` — read and write Q in the mega layout (32 chunks of `[heads, 16]` per token), which moves the padding heads to the tail of every head-dim chunk. A template parameter, because it selects the address arithmetic.

They are orthogonal, and that is tested. A lane owns dims `[16l, 16l+16)` of its head in **either** layout — only the address differs — so the per-head RMSNorm and the GPT-J RoPE are equally valid interleaved. `test_q_interleaved_is_orthogonal_to_norm_and_rope` pins it over all eight of (16, 64 heads) × (norm on/off) × (RoPE on/off): the head-major result permuted into the interleaved layout must equal the interleaved path on a permuted input, bit for bit, with the KV cache byte-identical so the Q layout cannot leak into the KV half. An earlier revision conflated the two into one flag on the mistaken belief that the layout made RoPE ill-defined; that test fails against the conflated form.

`q_head_padded = 0` additionally asks for the KV insert alone, which is what TP1 wants — its 64 live heads are already the kernel’s head count, so no Q pass is scheduled at all.

## One buffer pair per step

The output travels as a `QuantizedActivation` (`kMxfp8Dynamic`: fp8 e4m3 data, packed-ue8m0 MN-major scales). FlashMLA’s fused ops take caller-provided `out_fp8` / `out_sf`, so a single buffer pair covers a whole step: the prefill chunks and the decode segment fill disjoint token ranges, and one `fp8_einsum` over all N tokens replaces one call per segment. The scale buffer’s head-dim stride is `ceil4(N)` — both what the kernel requires of a caller buffer and what DeepGEMM expects for that N.

`wq_b` rows and `wo_a` columns are permuted once at load into the kernel’s chunk-interleaved layouts, so the surrounding GEMMs speak them directly rather than shuffling per step. DSpark’s draft layers are ordinary `DeepseekV4DecoderLayer`s and therefore take the engine-wide backend, so they owe the same post-load permutation; `DSparkDeepseekV4ForCausalLM.process_weights_after_loading` now runs it, where before it finalized only MoE weights and every draft forward hit the layer’s own guard.

## NVFP4 compressed cache

`nvfp4_ds_mla` now means the V4.1 MXFP8 sliding-window record beside a 288 B NVFP4 compressed record (256 B of e2m1 pairs, then 32 e4m3 scales of 16 dims). It rides the byte-width dispatch the fp8 records already use, so `rope_quant_insert` and `dequantize_and_gather_k_cache` pick it up with no call-site change; CuteDSL has no fp4 gather, so that record takes Triton.

This is the format the reference implementation itself stores: `inference/model.py` quantizes the compressed latent with `fp4_act_quant(latent, 16, True, scale_dtype=torch.float8_e4m3fn)` — group 16, e4m3 scales. `fp8_ds_mla` is _more_ precise than the reference on that cache, not less.

Each layer declares the packed record its kernel prefers, and an unspecific `--kv-cache-dtype` resolves to it:

| layer                   | `auto` / `fp8` | explicit `fp8_ds_mla` | explicit `nvfp4_ds_mla` |
| ----------------------- | -------------- | --------------------- | ----------------------- |
| mega attention          | `nvfp4_ds_mla` | `fp8_ds_mla`          | `nvfp4_ds_mla`          |
| `FLASHMLA_SPARSE_DSV41` | `fp8_ds_mla`   | `fp8_ds_mla`          | rejected by the backend |
| plain-row (FlashInfer)  | `fp8`          | —                     | `ValueError`            |

## Default backend on SM100

An unspecified `--attention_config.backend` now resolves to `FLASHMLA_MEGA_ATTN_DSV41` on SM100. The layer answers `is_available_for()` rather than letting the selector guess, because `__init__` raises on topologies the kernel cannot serve — it is SM100 only, it needs `WV_GROUP_SIZE` heads per `wo_a` group, and a build may not carry the kernel at all. On a 64-head model that admits TP1 through TP8 and declines TP16 (4 local heads), which falls through to `FLASHMLA_SPARSE_DSV41` instead of failing at startup.

The default is chosen for TP1’s win plus the 45%-smaller compressed record, **not** for a uniform speedup — see the table below, where TP2 through TP8 are a wash. Set `--attention_config.backend=FLASHMLA_SPARSE_DSV41` to get the previous path.

## Performance

`benchmarks/kernels/benchmark_dsv41_mega_attn.py`, one GB300, CUDA graphs on, counting every per-step op each path runs (topk_swa=128, topk_extra=512, NVFP4 compressed cache), as mega / `FLASHMLA_SPARSE_DSV41` microseconds on a 64-head model:



```
  s_q                    1      32     128     256     512
  TP1 (64 live heads)  29/29   31/31   31/40   53/72   96/140    up to 1.45x
  TP2 (32 live)        29/29   31/30   35/35   61/63  110/119
  TP4 (16 live)        29/29   31/29   34/34   59/58  109/109
  TP8 (8 live)         29/29   31/29   33/33   59/57  107/104
```

The decisive variable is live heads over padded heads, not batch size: the kernel absorbs the inverse RoPE and the FP8 cast, whose cost scales with _live_ heads, while its own cost scales with _padded_ heads — always 64, since FlashMLA’s FP8 decode takes only `h_q` in \{64, 128\}. So TP1 wins and TP2 down cancel to within a few percent. There is no batch size at which falling back pays, so there is no minimum-token threshold and no fallback path to maintain.

An earlier revision of this table also charged the mega arm for a separate Q-pad kernel “that the split-KV path gets for free”; that is no longer true — the pad is now inside the same fused insert both paths run, and at TP1 it is skipped entirely.

These are decode numbers; **prefill is unmeasured**. Measure under CUDA graphs — in eager mode launch overhead swamps both paths and overstates the fused one several-fold.

## Accuracy: measured end to end, with DSpark

The earlier revision listed accuracy as not covered. It now is. Both arms on one GB300, TP4, `deepseek-ai/DeepSeek-V4.1-Flash` at pinned snapshot `517ef625`, identical eval config, DSpark k=5 with **real** rejection sampling (`rejection_sample_method=synthetic` deliberately absent, so acceptance measures the draft rather than echoing the config). The two recipes differ by exactly one engine flag.

|                              | mega attention (NVFP4) | `FLASHMLA_SPARSE_DSV41` (fp8_ds_mla) | history band                     |
| ---------------------------- | ---------------------- | ------------------------------------ | -------------------------------- |
| gsm8k flexible / strict      | **0.9318** / 0.9318    | 0.9265 / 0.9265                      | median 0.9287, IQR 0.9265–0.9318 |
| gpqa_diamond exact / pass\@4 | **0.9053** / 0.9545    | 0.9066 / 0.9545                      | median 0.9040, IQR 0.8990–0.9141 |
| draft acceptance gpqa        | 0.4376 (len 3.188)     | 0.4341 (len 3.171)                   |                                  |
| draft acceptance gsm8k       | 0.4957 (len 3.479)     | 0.4945 (len 3.473)                   |                                  |

All four scores land inside their interquartile range, and the two arms agree within sampling noise (gsm8k 0.53 pt, gpqa 0.13 pt; gpqa is n=198 at temperature 1.0, so one question is \~0.5 pt). **The fused kernel and the 45%-smaller compressed record cost nothing measurable in accuracy.**

Draft acceptance is the more sensitive check, because it reads draft-versus-target agreement directly: all four figures span 0.4341–0.4376 on gpqa and 0.4945–0.4957 on gsm8k, and they also reproduce the pre-rebase run to three decimals.

Scores were validated before being quoted: `num_request_errors: 0` on every eval, step durations plausible (530/114 s and 519/113 s), gsm8k > gpqa cross-task ordering, flexible − strict gap exactly 0.0000, and truncation of 3/792 gpqa and 1/1319 gsm8k rows so that answered-only accuracy tracks the headline (mega 0.9087 / 0.9332; sparse 0.9100 / 0.9279). GSM8K runs `api_type: completions` 5-shot greedy, so it bypasses the chat template and is not the deployed serving path; GPQA runs the deployed chat path with thinking on and the `deepseek_v41` reasoning parser.

## Rebase onto [#56441](https://github.com/aarnphm/aarnphm.github.io/issues/56441)

[#56441](https://github.com/aarnphm/aarnphm.github.io/issues/56441) added a KV-only context-insert op to the same translation unit. Two of its call sites needed updating for this series and **git reported no conflict on either**, because neither side edited those lines — they would have surfaced only at compile time: `launchFusedDeepseekV4Templated` gains a template parameter (`is_q_interleaved`) and a runtime one (`apply_q_rope`), and the KV-only op calls it. Both are fixed at the call site, with a comment noting that zero query heads means no Q slot is scheduled so neither knob is reachable there.

The two series had also independently arrived at the same `kNumHeadsQPadded == 0` grid fix (I contributed it to [#56441](https://github.com/aarnphm/aarnphm.github.io/issues/56441) during review); it collapsed to one.

## Test Plan & Result



```
pytest tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py \
       tests/kernels/test_compressor_kv_cache.py \
       tests/kernels/test_dsv41_mega_attn_layouts.py \
       tests/kernels/test_fused_indexer_q_rope_quant.py \
       tests/kernels/test_fused_inv_rope_fp8_quant.py     # 522 passed, 17 skipped
pre-commit run --files <changed files>                    # all hooks pass
```

The last two suites are there because the op this PR changes is also called by the **DSv4.0** paths (`deepseek_v4/attention.py`, its dspark, xpu); the two new parameters are defaulted so those call sites still bind, which is worth testing rather than assuming.

New tests cover the NVFP4 record byte-exactly against a reference quantizer derived from FlashMLA’s `tests/quant.py` (scales exact, values within half an e2m1 step, untouched rows and page padding preserved), an insert→gather round trip, the `wq_b` / `wo_a` permutations against a reference that permutes the activation instead, the interleaved Q pad (live elements bit-exact, padding heads zero, both grid variants, and the KV-insert-only launch), the norm/RoPE orthogonality described above, and graph-replay coverage that a captured insert refills its allocated Q buffer over changed input.

## Not duplicating

[#56893](https://github.com/aarnphm/aarnphm.github.io/issues/56893) (merged) landed the V4.1 MXFP8 KV record this builds on. [#56344](https://github.com/aarnphm/aarnphm.github.io/issues/56344) is the earlier flag-gated form of this same integration and is closed. [#56441](https://github.com/aarnphm/aarnphm.github.io/issues/56441) (merged) added a KV-only wrapper op in the same files; disjoint in intent, and the interaction is described above.

## Not yet covered

- **Prefill perf is unmeasured** — every number above is decode.
- **Non-SM100 arches** are untouched by design; the kernel has no SM90 instantiation and the default declines there.
- Two commits in this series do not build standalone (a call-site fix landed one commit later than the parameter it fixes), which matters only for `git bisect` across the series, not for the tip or a squash-merge.

## AI assistance

This change was developed with AI assistance (Claude Code). The backend, the interface capability, the NVFP4 record, the default-backend selection, the tests, the benchmark and the accuracy runs above were produced in that session and reviewed by me.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**foraxe** (2026-09-15T06:27:08Z):

Hi @zyongye Thanks for the updated integration! I opened a small follow-up against your branch: [zyongye/vllm#13](https://github.com/zyongye/vllm/pull/13). It moves query padding into captured input preparation.
In a bounded TP4/MRV2 check, short-prefill TTFT improved by about 8.7%; no decode speedup is claimed. Tests and results are included.

Would you please take a look when convenient?

**mergify** (2026-09-15T22:06:47Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zyongye.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**zyongye** (2026-09-16T06:17:26Z):

/ci run

**github-actions** (2026-09-16T06:17:43Z):

✅ Triggered [Buildkite CI #89275](https://buildkite.com/vllm/ci/builds/89275) for commit `9f6e18fe642a`.

