---
title: '58671'
pageLayout: default
slug: thoughts/inference/vllm-prs/58671
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/58671.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#58671](https://github.com/aarnphm/aarnphm.github.io/issues/58671) \[ROCm\]\[DSv4.1\] Paged MXFP4 sparse indexer on aiter’s MQA-logits kernel

[vllm-project/vllm#58671](https://github.com/vllm-project/vllm/pull/58671) · cagrikymk · MERGED · created 2026-09-25T06:22:52Z · merged 2026-09-29T06:37:58Z

## Summary

Adds a ROCm (gfx950) path for DeepSeek V4.1’s indexer with an MXFP4 K cache, built on aiter’s paged MXFP4 MQA-logits Gluon kernel. The CUDA path is unchanged. There are two modes:

- **Regular MXFP4** (`indexer_kv_dtype="mxfp4"`): dense logits over the whole context, read straight from the paged K cache.
- **Sparse MXFP4** (opt-in, also set `indexer_sparse_logits=True`): the candidate source takes block maxima from the same pass that writes its logits, and the consumer layers score only the candidate pool instead of the whole context. Steps whose contexts are too short for the pool to pay off keep the regular pass, so it helps mainly at long contexts.

Both modes:

- The logits kernel reads the preshuffled paged K cache in place, so no K gather is needed, prefill included.
- K is written to the cache by aiter’s `k_norm_rope_mxfp4_cache`, and Q is quantized by `q_rope_mxfp4_quant`.
- Each prefill chunk is packed into one launch through `query_start_loc`. Uniform spec-decode steps run as `next_n`-row sequences, with the decode schedule built once per step.
- 128-token KV pages are preferred.

In sparse mode, the candidate pool is resolved once per step and shared by every consumer layer.

Requires aiter with the paged MXFP4 MQA-logits kernel and the cache ops (merged to aiter main) ([ROCm/aiter#5834](https://github.com/ROCm/aiter/pull/5834))

## Tests

- `tests/kernels/attention/test_rocm_paged_mxfp4_indexer.py`
- `tests/v1/attention/test_rocm_paged_mxfp4_indexer_plan.py`

## Performance

## Benchmark: C=32, 128 prompts of 350K tokens, 350 out, spec. decoding = 5+1

| metric                 | tot_baseline | mxfp4_dense |      Δ | mxfp4_sparse |      Δ |
| ---------------------- | -----------: | ----------: | -----: | -----------: | -----: |
| total tok/s            |       19,225 |      22,796 | +18.6% |       24,863 | +29.3% |
| output tok/s           |        19.21 |       22.77 | +18.6% |        24.84 | +29.3% |
| requests/s             |       0.0549 |      0.0651 | +18.6% |       0.0710 | +29.3% |
| duration s             |        2,333 |       1,967 | -15.7% |        1,804 | -22.7% |
| TTFT mean s            |        378.2 |       319.4 | -15.5% |        292.3 | -22.7% |
| TTFT median s          |        400.8 |       339.3 | -15.4% |        309.0 | -22.9% |
| TTFT p99 s             |        556.9 |       473.2 | -15.0% |        436.7 | -21.6% |
| TPOT mean ms           |        490.8 |       412.1 | -16.0% |        380.5 | -22.5% |
| TPOT median ms         |        513.8 |       429.9 | -16.3% |        399.1 | -22.3% |
| TPOT p99 ms            |        638.6 |       525.8 | -17.7% |        467.4 | -26.8% |
| ITL median ms          |        816.2 |       695.0 | -14.8% |        638.7 | -21.7% |
| E2E mean s             |        549.5 |       463.3 | -15.7% |        425.1 | -22.6% |
| spec acceptance length |         1.64 |        1.65 |  +0.9% |         1.64 |  +0.3% |

Note, since this is done on artificial data, AL is lower (1.65) compared to more realistic 3.5.

8k/1k, NP=10\*C, TP4, using mxfp4 dense path (too short kv for sparse), no spec decoding:

|   C | FP8 output tok/s | MXFP4 output tok/s |         Δ | FP8 mean TTFT | MXFP4 mean TTFT |     Δ | FP8 mean TPOT | MXFP4 mean TPOT |         Δ |
| --: | ---------------: | -----------------: | --------: | ------------: | --------------: | ----: | ------------: | --------------: | --------: |
|   1 |            102.0 |              109.3 | **+7.2%** |        291 ms |          290 ms | -0.3% |       9.53 ms |         8.88 ms | **-6.8%** |
|  16 |           1027.9 |             1061.0 | **+3.2%** |       1611 ms |         1580 ms | -1.9% |      14.00 ms |        13.55 ms | **-3.2%** |
|  64 |           2005.7 |             2065.9 | **+3.0%** |       2048 ms |         2025 ms | -1.1% |      29.92 ms |        29.01 ms | **-3.0%** |
| 128 |           2485.5 |             2564.0 | **+3.2%** |       2835 ms |         2798 ms | -1.3% |      48.73 ms |        47.18 ms | **-3.2%** |

## Accuracy

| check                  |    tot_baseline |    mxfp4_sparse |     mxfp4_dense |
| ---------------------- | --------------: | --------------: | --------------: |
| GSM8K flexible-extract | 0.9651 ± 0.0051 | 0.9613 ± 0.0053 | 0.9598 ± 0.0054 |

### Long context accuracy check

- **The text:** the first 131,072 tokens of _War and Peace_ (Project Gutenberg), tokenized with the model’s own tokenizer. The token ids are saved, so every run scores exactly the same sequence.
- **One request** to the running server: `/v1/completions` with the ids as the prompt, `max_tokens=1`, `temperature=0`, `prompt_logprobs=0`. For every position the server returns the log-probability the model gave to the token that actually comes next, given everything before it. Nothing is sampled; it is one teacher-forced pass over the whole book.
- **The score:** NLL per token is minus that log-probability, averaged in four position buckets: 0-16K, 16K-32K, 32K-64K and 64K-128K.
- **Two runs per configuration** on the same server.

| run               | 0K-16K | 16K-32K | 32K-64K | 64K-128K |
| ----------------- | -----: | ------: | ------: | -------: |
| tot_baseline run1 | 0.0710 |  0.1302 |  0.1323 |   0.1460 |
| tot_baseline run2 | 0.0717 |  0.1293 |  0.1332 |   0.1484 |
| mxfp4_sparse run1 | 0.0709 |  0.1274 |  0.1334 |   0.1465 |
| mxfp4_sparse run2 | 0.0711 |  0.1269 |  0.1300 |   0.1449 |
| mxfp4_dense run1  | 0.0715 |  0.1313 |  0.1353 |   0.1489 |
| mxfp4_dense run2  | 0.0721 |  0.1302 |  0.1330 |   0.1494 |

## Not a duplicate

[#57517](https://github.com/aarnphm/aarnphm.github.io/issues/57517) and [#56834](https://github.com/aarnphm/aarnphm.github.io/issues/56834) also add a ROCm MXFP4 indexer K cache, with dense scoring. The main
addition here is sparse MXFP4 (`indexer_sparse_logits=True`): the candidate source takes
block maxima from the same pass that writes its logits, and the consumer layers score only
the candidate pool, read in place from the paged MXFP4 cache. Neither PR supports this
([#56834](https://github.com/aarnphm/aarnphm.github.io/issues/56834) rejects candidate blocks outright). For the dense path, this PR uses aiter’s paged
MXFP4 MQA-logits kernel ([ROCm/aiter#5834](https://github.com/ROCm/aiter/issues/5834), in aiter v0.1.23) and leaves vLLM’s shared Q/K
kernels untouched.

Follows the main idea from [vllm-project/vllm#56254](https://github.com/vllm-project/vllm/pull/56254) while going direct to the paged K for prefill.

## AI assistance

Claude is used during development, and the PR author reviewed the AI driven changes.

---

## comments

**mergify** (2026-09-27T16:58:48Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @cagrikymk.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**github-actions** (2026-09-29T01:07:00Z):

✅ @cagrikymk, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**mergify** (2026-09-29T01:14:39Z):

Hi @cagrikymk, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-09-29T01:24:43Z):

Hi @cagrikymk, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**cagrikymk** (2026-09-29T02:36:45Z):

/ci run

**github-actions** (2026-09-29T02:37:04Z):

❌ This PR is 1 commit behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5882572620 -->

**shen-shanshan** (2026-09-29T02:39:23Z):

/ci run

**github-actions** (2026-09-29T02:39:41Z):

✅ Triggered [Buildkite CI #91719](https://buildkite.com/vllm/ci/builds/91719) for commit `cd39d3d3672a`.

**cagrikymk** (2026-09-29T04:42:48Z):

/ci run

**github-actions** (2026-09-29T04:43:08Z):

✅ Triggered [Buildkite CI #91744](https://buildkite.com/vllm/ci/builds/91744) for commit `8fc68e0c64bb`.

