---
title: '56893'
pageLayout: default
slug: thoughts/inference/vllm-prs/56893
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56893.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56893](https://github.com/aarnphm/aarnphm.github.io/issues/56893) \[Model\]\[DSv4.1\] Store the whole KV in MXFP8 (FlashMLA V4.1 record)

[vllm-project/vllm#56893](https://github.com/vllm-project/vllm/pull/56893) · zyongye · MERGED · created 2026-09-14T20:31:07Z · merged 2026-09-15T03:12:56Z

## Purpose

DeepSeek-V4.1 has been reusing DeepSeek-V4’s paged `fp8_ds_mla` record. FlashMLA has a V4.1-specific one that quantizes the RoPE dims too; this switches DeepSeek-V4.1 onto it on SM100.

|                   | V4 (`ModelType::V4`)                  | **V4.1 (`ModelType::V41`)**           |
| ----------------- | ------------------------------------- | ------------------------------------- |
| data row          | 448 B fp8 e4m3 NoPE + 128 B bf16 RoPE | **512 B fp8 e4m3, RoPE included**     |
| scale row         | 7 UE8M0 of 64 dims + 1 pad byte       | **16 UE8M0, one per 32 dims (MXFP8)** |
| bytes/token       | 584                                   | **528**                               |
| page (TMA) stride | 576 B                                 | **512 B**                             |

Both records put a page’s whole data region ahead of its whole scale region, so a page is `block_size * bytes_per_token` rounded up to the TMA stride. Because `block_size` is always a multiple of 32, every V4.1 page is an exact multiple of 512 — the padding the V4 record needed disappears, on top of the 9.6% smaller record.

Gated on SM100: `ModelType::V41` is absent from FlashMLA’s SM90 sparse-decode feature set, so SM90, ROCm and the FlashInfer SM120 path keep the V4 record. No new cache dtype and no new flag — `--kv-cache-dtype fp8` resolves to `fp8_ds_mla` as before and the record follows the arch, exactly as `fp8_ds_mla` already means 656 B for V3.2 and 584 B for V4.

### What changed

- **FlashMLA pin** → `vllm-project/FlashMLA@b6ef01c`, following the fork’s tree reorganization (`csrc/sm*` → `csrc/kernels/sm*`, API implementations split into `.cpp` files). `api.cpp` now registers `fused_norm_rope_attn_rope_cast_*` and the two permute kernels, so their instantiations must be compiled for `_flashmla_C` to link; the source list mirrors the fork’s `setup.py` minus the dense backward, which is training-only.
- **Writers**: the fused SWA insert takes a `kv_mxfp8` argument (one UE8M0 scale per lane pair instead of per four, and the RoPE lanes quantize instead of storing bf16); the compressed-cache insert gets an MXFP8 kernel that applies RoPE _before_ choosing scales, since the RoPE dims are now quantized.
- **Readers**: the Triton gather and the CuteDSL gather are parameterized by record instead of hard-coded to 448/64, and both pick it from `k_cache.shape[-1]` — the per-token width the KV cache spec already publishes — so the V4 record keeps working unchanged on the same code path.

### The indexer page has to move with them

`_get_bytes_per_block` makes a block’s stride the **sum** of every page in the group, and DeepSeek-V4/V4.1 packs the SWA cache, the compressed-KV cache, the indexer cache and the compressor ring into one block. Divisibility by the TMA stride therefore only holds if _every_ contributing page carries the same padding — which is what the indexer cache’s 576 B alignment was for, rather than any constraint of its own kernel.

Leaving the indexer at 576 while the main caches moved to 512 produced a block stride of 212160 (a multiple of neither) and tripped FlashMLA’s `block_stride_bytes % TMA_K_STRIDE == 0` assertion in the startup profile run, after a clean weight load. Every per-spec page size looked correct in isolation. This PR ties the indexer’s alignment to the same gate.

## Relationship to other open PRs

- **[#56344](https://github.com/aarnphm/aarnphm.github.io/issues/56344) (`[Perf] Deepseek V41 Attention MegaKernel`, same author)** integrates FlashMLA PR [#221](https://github.com/aarnphm/aarnphm.github.io/issues/221)’s fused norm+RoPE+attention megakernel behind `--attention-config '{"dsv4_fused_attention": true}'` and adds an `nvfp4_ds_mla` dtype pairing a 528 B SWA cache with a 288 B fp4 compressed cache. This PR is the narrower, unconditional half: it moves DeepSeek-V4.1’s _existing_ `fp8_ds_mla` cache to the V4.1 record with no opt-in and no new dtype, and it does not touch the attention path. The two overlap on `flashmla.cmake` (that one pins `c112cc1`, this one `b6ef01c` — one commit later, a test-only change in the fork) and on the V4.1 writers/readers; whichever lands first, the other rebases onto it. [#56344](https://github.com/aarnphm/aarnphm.github.io/issues/56344) does not currently touch `DeepseekV4IndexerCache.get_kv_cache_spec`, so the alignment fix above is worth carrying across either way.
- **[#56441](https://github.com/aarnphm/aarnphm.github.io/issues/56441) (`[Perf][DSpark] Add KV-only context insertion across V4.1 cache formats`)** touches the same three `csrc/libtorch_stable/` files to add a KV-only wrapper op. The changes are disjoint in intent (its wrapper drops the dummy query; this adds a record selector) but will conflict textually on the op signature and bindings.

Not duplicating either: neither makes the V4.1 MXFP8 record the layout DeepSeek-V4.1 actually runs with by default, and neither carries the indexer-alignment fix.

## Test Plan & Result



```
pytest tests/kernels/test_compressor_kv_cache.py \
       tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py   # 271 passed, 11 skipped
pytest tests/v1/attention/test_sparse_mla_backends.py                 # 392 passed, 378 skipped
pytest tests/kernels/attention/test_flashmla.py \
       tests/kernels/attention/test_flashmla_sparse.py                # 10 passed, 144 skipped
pytest tests/kernels/attention/test_cache.py -k mla \
       tests/model_executor/layers/test_mla_short_prefill_indexer.py \
       tests/kernels/test_cp_gather_fp8.py                            # 97 passed, 12 skipped
pytest tests/v1/worker/test_dsv4_packed_zeroer_geometry.py \
       tests/v1/attention/test_indexer_deepseek_v4_slot_mapping.py    # 20 passed
pre-commit run --files <changed files>                                # all hooks pass
```

Three new tests cover the record: a byte-exact check of the compressed-cache insert against a reference quantizer derived from FlashMLA’s `tests/quant.py`, an insert→gather round trip (which also exercises the CuteDSL path), and the CUDA op’s MXFP8 branch against the Triton encoder.

## Model evaluation

DeepSeek-V4.1-Flash (revision `517ef625`), TP4 on one GB300, `--kv-cache-dtype fp8 --block-size 128 --max-model-len 65536`, no speculative decoding. Baseline is the same weights/TP/node on the V4 584 B record.

| task                                     | V4 record (584 B)       | **V4.1 MXFP8 (528 B)**                       |
| ---------------------------------------- | ----------------------- | -------------------------------------------- |
| gsm8k, greedy 5-shot completions, n=1319 | 0.9272 / 0.9310         | **0.9333** flexible · **0.9340** strict      |
| gpqa_diamond, thinking, T=1.0            | 0.904 / 0.899 (epoch 1) | **0.9028** (epoch 4, n=792) · pass\@4 0.9495 |

0 request errors on both. GPQA truncated 3/792 rows; answered-only accuracy 0.9062. Quantizing the 64 RoPE dims costs nothing measurable.

Comparability caveat: the baselines were taken with the `deepseekv41-flash-0909` image at gpqa `epoch_num: 1`, this run with a local build at `epoch_num: 4`, so the arms differ in image and epoch count as well as in the record. The gsm8k arm at temperature 0 is the tight comparison and it is clean.

Engine-level evidence beyond the scores: FlashMLA hard-errors on any byte count that is not 584/528/288 for `d_qk=512`, so a clean startup — memory profiling, KV allocation (161.73 GiB, 11,589,848 tokens) and full CUDA-graph capture of the sparse decode path — is itself proof the 528 B record was accepted as `ModelType::V41`.

## AI assistance

This change was developed with AI assistance (Claude Code). The FlashMLA pin bump, the record change across the writers/readers, the indexer-alignment fix, the new tests and the evaluation runs above were produced in that session and reviewed by me.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**zyongye** (2026-09-14T21:19:25Z):

/ci run

**github-actions** (2026-09-14T21:19:46Z):

✅ Triggered [Buildkite CI #88922](https://buildkite.com/vllm/ci/builds/88922) for commit `a1c5eecb69f1`.

**zyongye** (2026-09-14T21:58:44Z):

/ci run

**github-actions** (2026-09-14T21:59:03Z):

✅ Triggered [Buildkite CI #88936](https://buildkite.com/vllm/ci/builds/88936) for commit `c0a11da0acab`.

**zyongye** (2026-09-15T01:29:38Z):

/ci run

**github-actions** (2026-09-15T01:29:55Z):

❌ This PR is 9 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5673286243 -->

**zyongye** (2026-09-15T01:30:25Z):

/ci run

**github-actions** (2026-09-15T01:30:43Z):

✅ Triggered [Buildkite CI #88975](https://buildkite.com/vllm/ci/builds/88975) for commit `38465d553f40`.

