---
title: '56254'
pageLayout: default
slug: thoughts/inference/vllm-prs/56254
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56254.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56254](https://github.com/aarnphm/aarnphm.github.io/issues/56254) \[DSA\] Wire DeepGEMM sparse MQA logits into the DeepSeek V4.1 indexer

[vllm-project/vllm#56254](https://github.com/vllm-project/vllm/pull/56254) · JaredforReal · MERGED · created 2026-09-10T10:27:08Z · merged 2026-09-15T21:07:15Z

## Purpose

DeepSeek V4.1 selects attention tokens in two levels: one indexer layer publishes the top candidate blocks of the context, and the indexer layers after it pick their top-k inside those blocks. Today the consumer layers still score the **whole** context with dense MQA logits and then mask everything outside the candidates. DeepGEMM 2.8 ships sparse MQA-logits kernels ([deepseek-ai/DeepGEMM#432](https://github.com/deepseek-ai/DeepGEMM/issues/432)) that score only a given list of blocks, so this PR lets the consumer layers score just the candidate blocks. Their logits work goes from O(context) to O(candidate blocks), a fixed 16K positions for V4.1-Flash, and their top-k runs in that compressed space and is mapped back to context positions.

Design:

- **A separate indexer, not a branch in the dense one.** The consumer layers use a new `SparseMQAIndexer` layer with its own backend and attention metadata (`DeepseekV41SparseIndexerBackend`, `DeepseekV41SparseIndexerMetadata`). It is a plain module rather than a custom op: V4.1 does not go through `torch.compile`, and the attention layer already runs the indexer inside its breakable-CUDA-graph eager segment. The dense `SparseAttnIndexer` op is untouched. The candidate-source layer and the indexers before it stay dense.
- **Every requirement is checked once, at engine start.** The metadata builder validates the model, the MXFP4 indexer cache, the GPU generation, the DeepGEMM version, the DeepSelect extension and the logits/top-k layout it needs, and raises a clear error instead of silently falling back per step. There is no PyTorch fallback anywhere on the path: the candidate expansion and the index remap are Triton kernels, the top-k is DeepSelect.
- **Nothing is allocated or recomputed on the hot path.** The builder owns the per-row scratch; the candidate-block expansion and the DeepGEMM schedule depend only on the candidates, which are identical for all consumer layers of a step, so the first consumer computes them and the others reuse them through the shared metadata.
- **No dtype casts on the hot path.** The sparse kernels take bf16 weights and emit bf16 logits. The fused Q RoPE-quant kernel that produces the per-head weights writes them in bf16 for this indexer (fp32 for the dense one), and DeepSelect ([#56464](https://github.com/aarnphm/aarnphm.github.io/issues/56464)) runs the top-k on the bf16 logits directly.
- **Opt-in** via `--attention-config '{"indexer_kv_dtype": "mxfp4", "indexer_sparse_logits": true}'`. Below \~16K tokens the candidates cover the whole context and the dense path is cheaper, so the default stays off; the feature is for long-context serving.
- **One KV-layout requirement.** DeepGEMM’s paged kernels address pages as `base + page * stride` and need that stride 512B-aligned. In V4.1’s block-major layout the stride is the whole block (all layers’ pages), which is not 512-aligned. `MLAAttentionSpec` gains `block_stride_alignment`, and the block size is rounded up to it in the same place HiSparse already rounds it (`_get_kv_cache_bytes_per_block`). With the flag on this costs 0.8% of KV memory; with it off nothing changes.

## Test Plan

- `tests/model_executor/layers/test_mla_short_prefill_indexer.py`: candidate-expansion math and a randomized comparison of the Triton expansion against a PyTorch reference (including strided inputs); exact selected-value parity of the sparse path with `torch.topk` over dense bf16 logits masked to the candidates, for prefill and paged decode, including spec-decode rows, CUDA-graph capture/replay and schedule reuse by a later layer; kernel numerics vs a PyTorch fp32 reference. `tests/kernels/test_fused_indexer_q_rope_quant.py`: the bf16 weights output of the fused Q kernel (Triton and CuTeDSL) equals the fp32 result rounded once. mypy and ruff via pre-commit.
- Kernel microbenchmark of the per-layer indexer pipeline (dense logits + mask + top-k vs sparse logits + top-k + remap) with an exact selected-value parity check, GB300, CUDA-graph replay.
- DeepSeek-V4.1-Flash, TP=4 on 4x GB300, MXFP4 indexer cache, greedy, prefix caching off, flag on vs off: decode and prefill throughput from 8K to 1M context, and long-context NLL.

## Test Result

Unit tests: 28 passed. The sparse kernels produce bitwise the same bf16 logits as the dense kernels on the candidate columns, so the tests assert that the sparse path selects exactly the same values per row as `torch.topk` over dense bf16 logits masked to the candidates; only tie order among equal values may differ.

**Kernel level** (GB300, speedup of the sparse indexer pipeline over the dense one per consumer layer; decode rows are one query per request, the prefill column is a chunk of query rows sharing their sequence’s K. V4.1-Flash has 4 consumer layers, the first one also pays the once-per-step candidate expansion, which lowers its speedup by 20-50%):

| context | decode, batch 64 | decode, batch 256 | prefill chunk |
| ------- | ---------------: | ----------------: | ------------: |
| 8K      |             1.2x |              1.2x |          1.6x |
| 32K     |             1.7x |              1.8x |          2.3x |
| 128K    |             4.4x |              5.8x |          8.3x |
| 512K    |            14.4x |             22.9x |               |

**End to end** (DeepSeek-V4.1-Flash, TP=4, 4x GB300; decode and prefill throughput of the whole batch, from a linear fit of wall time against output length):

| batch x context | decode tok/s off | decode tok/s on | decode speedup | prefill tok/s off | prefill tok/s on | prefill speedup |
| --------------- | ---------------: | --------------: | -------------: | ----------------: | ---------------: | --------------: |
| 64 x 8K         |            5,570 |           5,895 |          1.06x |            68,712 |           65,541 |           0.95x |
| 32 x 100K       |            3,263 |           3,434 |          1.05x |            60,508 |           64,306 |           1.06x |
| 16 x 256K       |            2,055 |           2,153 |          1.05x |            48,504 |           57,670 |           1.19x |
| 8 x 512K        |            1,057 |           1,107 |          1.05x |            34,127 |           48,805 |           1.43x |
| 4 x 1M          |              568 |             583 |          1.03x |            16,261 |           32,517 |           2.00x |

Decode gets 3-6% faster: the indexer is only a few percent of a decode step at these batch sizes. Prefill is where the O(context) cost shows: the dense indexer’s fp32 logits grow with the context and are chunked, written and read back several times, while the sparse rows stay 16K wide, so prefill throughput is 1.2x at 256K, 1.4x at 512K and 2x at 1M, while at 8K it is 5% slower.

**Accuracy** (78K-token real document, mean NLL of the next token by position; the sparse kernels score in bf16, the dense path in fp32). Two runs of the same configuration differ by up to 0.002 per bucket (TP=4 run-to-run noise), and on/off differ by the same amount:

| position       |  0-16K | 16K-32K | 32K-64K | 64K-78K |
| -------------- | -----: | ------: | ------: | ------: |
| off            | 0.4001 |  0.3715 |  0.3523 |  0.3735 |
| on             | 0.4022 |  0.3706 |  0.3526 |  0.3755 |
| on, second run | 0.4010 |  0.3719 |  0.3534 |  0.3737 |

## Requirements and limitations

- SM100-class GPUs, DeepGEMM >= 2.8 (the repo pin is older; the flag fails at startup with a clear message on older builds), the DeepSelect extension (`vllm._deepselect_C`, built with vllm) and the MXFP4 indexer cache (the kernels take UE8M0 scales). `--sparse-indexer-topk-backend` must be `auto` or `deep_select`. Not supported with DCP/PCP.
- The sparse kernels score in bf16 with bf16 weights (no fp32 option), so the flag changes the indexer’s score precision; greedy outputs are not token-identical to the dense path (see the NLL table).

AI assistance: this change was developed with an AI coding agent; the human submitter reviewed the full diff and ran the tests above.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**mgoin** (2026-09-10T17:32:12Z):

We can add support for BF16 input to the persistent topk like shown in [vllm-project/vllm#40811](https://github.com/vllm-project/vllm/pull/40811) now that we natively get it from deepgemm, avoiding the unneeded cast

**zyongye** (2026-09-11T05:46:19Z):

Let’s create another sparse indexer class with it’s own backend and attention metadata.

**ywang96** (2026-09-12T08:39:56Z):

Thanks for the unusually well-documented PR — the cost-model note, benchmarks, and accuracy data made this straightforward to review (reviewed at [`16a2fee`](https://github.com/aarnphm/aarnphm.github.io/commit/16a2fee)). The approach (score only candidate blocks, top-k in sparse space, remap to request-local) is sound, the gating is fail-closed, and the math checks out. **Verdict: approve with comments** — one must-fix below (import-time crash on Triton-less platforms), the rest is non-blocking.

## Correctness findings

**Major**

1. **Must-fix: import-time crash on Triton-less platforms** — `vllm/model_executor/kernels/attention/dsa/sparse_mqa_logits.py:41`: `_INT32_BIG = tl.constexpr(2**31 - 1)` calls `tl.constexpr` at module scope. When Triton is unavailable, `vllm.triton_utils` substitutes `TritonLanguagePlaceholder`, where `constexpr = None` (not callable), so this raises `TypeError` at import time — and `sparse_attn_indexer.py:22-31` imports this module unconditionally, so merely importing the indexer layer (e.g. on CPU builds) breaks. The sibling `candidate_blocks.py` uses `tl.constexpr` only in annotations, never as a module-level call. One-line fix: `_INT32_BIG = 2**31 - 1` (plain ints captured as globals work fine inside `@triton.jit`); that also unifies the sentinel with the `torch.iinfo(torch.int32).max` the fallback path uses at :232.

2. **Sparse prefill can exceed the profiled logits memory budget** — `sparse_mqa_logits.py:293` (`logits.float()`) makes the sparse prefill path transiently hold bf16 logits **plus** an fp32 copy of `[rows, num_candidates × cbk]`, plus a private 1 MiB radix workspace (:298) and sparse indices — all allocated outside `current_workspace_manager()`. Chunk sizing and the dummy-run reservation assume `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB` (512 MiB, `sparse_attn_indexer.py:377-382`), but the metadata builder doesn’t know the sparse path is active: an 8192-row chunk × 16384 sparse columns is \~256 MiB bf16 + \~512 MiB fp32 ≈ 768 MiB transient. In memory-tight deployments this can OOM where the dense path would not. Suggest avoiding the fp32 cast (fused cast into top-k, or chunked top-k), routing the workspace through the manager, or shrinking chunks when `use_sparse_mqa` is on.

**Minor**

3. Sparse decode always uses `persistent_topk` (`sparse_mqa_logits.py:300`) and never the `cooperative_topk` the dense path prefers for `num_rows <= 64` (`sparse_attn_indexer.py:758-770`). Likely part of why sparse loses at ≤32K context in your table; fine for a long-context flag, but worth a TODO.
4. Per-step allocations in the hot path: the 1 MiB radix workspace is `torch.empty`’d every decode step (:298) instead of via the workspace manager the dense path uses, plus `torch.zeros(rows)` per prefill chunk (:304).
5. Numerics: the sparse kernels take bf16 weights and emit bf16 logits while the dense path uses fp32 weights/logits (`deep_gemm.py:555,571` vs the new wrappers’ bf16 requirement). Inherent to the DeepGEMM kernels and covered by your overlap/e2e data — a one-line note in the `VLLM_DSA_SPARSE_MQA_LOGITS` env help would set expectations.
6. CI coverage: all new GPU tests skip without SM100 + DeepGEMM ≥ 2.8, and the repo pin is older, so the new kernels are untested in CI until the pin bump you list as a prerequisite. Consider landing the pin bump (or a tracking issue) alongside this.
7. Capture mode in the paged test compares only replay-vs-eager value multisets, never against the dense reference (`tests/model_executor/layers/test_mla_short_prefill_indexer.py:726-745`); non-capture mode does cover it, so this is about the test’s own symmetry.

<details>
<summary>Traced and verified OK (no action needed)</summary>

- Candidate→sparse-block expansion math: unaligned-ks anchoring, partial tail blocks, empty rows, ±2^30 garbage/overflow, repeat-last padding convention; remap formula `pos = (si[j] − ks//SBK)·SBK + o`; valid-columns-are-a-prefix invariant used by the radix top-k via `end`.
- `-1` tail when `k < topk_tokens` is safe via the pre-cleared `topk_indices_buffer` (`sparse_attn_indexer.py:468-474`).
- DCP safety: entry assert `dcp_world_size == 1` for candidates, decode gated on `global_seq_lens is None`, `_merge_dcp_topk_global` early-returns at world size 1 — the prefill `continue` skips a no-op.
- Paged-layout fallbacks (512B page stride, `page_kv % sbk`, block-table `repeat_interleave` order for MTP) fail closed to the dense path.
- Fail-closed DeepGEMM bindings; CUDA-graph capture/replay exercised by tests.

</details>

## Style findings

**Minor**

1. Duplicated constants across the layer/kernels boundary: `_RADIX_TOPK_WORKSPACE_SIZE` (`sparse_mqa_logits.py:39`) vs `RADIX_TOPK_WORKSPACE_SIZE` (`sparse_attn_indexer.py:57`), and `_TOPK_KERNEL_SUPPORTED` (:36) vs the inline `(512, 1024, 2048)` tuples at `sparse_attn_indexer.py:82,760,767`. Two sources of truth that must stay in sync — since imports flow layer→kernels, define them once in the kernels module and import from there.
2. Magic number `512` for the page-stride alignment (`sparse_mqa_logits.py:435`, repeated in the test at :659) — a named constant next to the wrapper docstrings would help.
3. Dead code in the capture test: `for sel in (out, eager_out): pass` (`test_mla_short_prefill_indexer.py:735-736`). Ruff won’t flag it (B007 is disabled repo-wide), so it needs a manual delete.
4. Redundant double slice in `_make_candidates` (:415): `randperm(a)[:n].sort().values[:num_candidates]` — the second slice is a no-op since `n = min(a, num_candidates)`.

**Nit**

5. `rows68` (:624) — a named `row_bytes = head_dim // 2 + 4` would make the fused-layout packing readable without reverse-engineering.
6. `sparse_mqa_logits_paged_decode(...) -> bool` — a `try_` prefix would signal the fallible “False if unsupported” contract at the call site.

Checked and clean: `ruff check` and `ruff format --check` (repo-pinned v0.14.0 with the repo `pyproject.toml`) pass on all touched files; SPDX header present; new module placement next to `candidate_blocks.py` keeps the layer file lean; the `envs.py` flag follows the two-place annotation+lambda pattern; the new `deep_gemm.py` wrappers faithfully mirror the existing lazy-init / `_missing()` fail-closed idiom.

---

**Merge-blocking:** only correctness finding 1 (the module-level `tl.constexpr` call) — a one-line fix. Finding 2 (memory accounting) should be addressed or explicitly documented before the flag flips from opt-in to default; everything else is follow-up-able. Nice work.

**mergify** (2026-09-12T15:09:22Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @JaredforReal.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-15T03:50:23Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @JaredforReal.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**WoosukKwon** (2026-09-15T08:18:22Z):

As we discussed offline, I’m ok to merge the PR once those above reviews are addressed. Great work!

**mergify** (2026-09-15T11:47:14Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @JaredforReal.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**JaredforReal** (2026-09-15T12:04:02Z):

clean up done, a bit unsure about the intro of `weights_out_dtype` in dsv4 ops
PTAL, thanks @WoosukKwon @ywang96

**WoosukKwon** (2026-09-15T18:08:05Z):

/ci run

**github-actions** (2026-09-15T18:08:25Z):

✅ Triggered [Buildkite CI #89145](https://buildkite.com/vllm/ci/builds/89145) for commit `3ac96f2a4b97`.

