---
title: '60397'
pageLayout: default
slug: thoughts/inference/vllm-prs/60397
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/60397.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#60397](https://github.com/aarnphm/aarnphm.github.io/issues/60397) \[ROCm\]\[Perf\]\[DSv4.1\] Mono decode layer for DeepSeek-V4.1-Flash on MI355X

[vllm-project/vllm#60397](https://github.com/vllm-project/vllm/pull/60397) · Fangzhou-Ai · MERGED · created 2026-10-07T13:29:15Z · merged 2026-10-10T01:53:19Z

## Purpose

Opt-in (`VLLM_ROCM_MONO_DECODE=1`, meant as the one switch for every model’s mono decode kernels) fast path for DeepSeek-V4.1-Flash decode on MI355X (gfx950). Every backbone layer of a decode step runs on persistent FlyDSL launches:

| Launch | Layers             | Stages                                                                                                                                                                            |
| ------ | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| K1     | 30 standard layers | the mHC attention seam, then the attention front: wqkv, q/kv norms, the KV insert, wq_b                                                                                           |
| K2     | 30 standard layers | the attention back (split attention, combine, wo_a, wo_b), the attention TP all-reduce, the FFN seam, the MoE (router, top-6, shared expert, MXFP4 experts) and its TP all-reduce |
| FFN    | the other 10       | after vLLM’s attention seam and attention: the attention TP all-reduce (wo_b’s unreduced output pushed to every rank), then K2’s FFN seam and MoE                                 |

All TP all-reduces run in-kernel over symmetric peer memory. K1 + K2 replace the \~40 kernels this path launches per standard layer today; the FFN launch replaces the attention all-reduce, the FFN seam and the MoE’s kernels on the others.

**Scope**

- **Layers:** K1 + K2 on the 30 backbone layers with the standard mHC seam and no compressor or indexer of their own: 3–7, 9–13, 15–19, 21–23, 25–27, 29–31, 33–35, 37–39. The FFN launch on the other 10: layer 0 (embedding-broadcast seam), the Engram layer 1, and the compressor/indexer layers 2, 8, 14 (also Engram), 20, 24, 28, 32, 36.
- **Steps:** decode only, ≤ 48 rows (8 requests × 1 + 5 DSpark drafts), TP2 or TP4, eager or a FULL CUDA graph. K1 + K2 also need causal SWA windows and `fp8_ds_mla` KV records.
- **Everything else** runs as before: the DSpark draft layers, and prefill, mixed or larger steps. The paths take and return the same tensors at the layer boundary, so they interleave freely.

### Layout



```
vllm/models/deepseek_v41/amd/mono_decode.py   eligibility per layer / per step, vLLM tensors -> kernel inputs
vllm/models/deepseek_v41/amd/mono/
    runner.py     DSV41MonoLayer: scratch, epoch, peer memory, the launches
    layer.py      K1 / K2 and the FFN launch, composed from the stages below
    attention/    attention stages
    stages/       seam.py (mHC seams, attention reduce), moe.py, moe_shape.py, moe_plan.py, router.py, gemv.py, dims.py
    common/       tagged mailbox, device ops, MX helpers, build keys, peer memory
```

`stages/` and `common/` are adapted from ROCm/ATOM’s DeepSeek-V4.1 mono decode (MIT License, commit [`97359d6`](https://github.com/aarnphm/aarnphm.github.io/commit/97359d6)). Each adapted file names its origin and keeps AMD’s copyright notice. The changes from ATOM:

- the seams are specialized to these layers;
- vLLM’s numerics: f32 router logits, the shared expert reading vLLM’s row-major FP8 weights, vLLM’s `mxfp8_e4m3_quantize` and `silu_and_mul_with_clamp`;
- hand-offs carry epoch tags instead of per-step mailbox clears, so no host step hook and the launches are graph-safe. ATOM’s clears also wiped what another step width had laid out differently; without them each width gets a scratch of its own;
- ATOM’s engine-side modules are dropped.

### Design

- **Weights** are read in place, with no second copy. The attention linears stay vLLM’s row-major MXFP8 (32×32 E8M0 scales). The routed experts stay in AITER’s shuffled MXFP4 layout.
- **Path selection.** The model is not torch.compiled, so the per-step decision is plain Python. It runs when a FULL graph is captured (uniform decode) and on every eager step. PIECEWISE (breakable) captures are always declined, because they are replayed for mixed batches.
- **wo_b’s reduce on the FFN-launch layers.** Their `wo_b` is built with `reduce_results=False`, as the sequence-parallel path already does. A decode step hands the partial to the FFN launch; any other step reduces it in the decoder layer with the same `tensor_model_parallel_all_reduce`. ROCm’s preshuffled-FP8 `wo_b` path now honors `reduce_results` as the linear does. This also fixes a bug on main: with sequence parallelism and an FP8-block checkpoint, `wo_b` was all-reduced and then reduce-scattered.
- **TP ranks stay in lockstep.** Every rank must take the same path at every layer, since the kernels wait on each other’s pushes. The conditions read only metadata that all ranks hold alike.
- **Failure modes.** An explicit opt-in that cannot run raises: a CDNA version outside the kernels’ (`get_cdna_version()` in `CDNA_VERSIONS`, CDNA4 / gfx950 today, so a later generation is one entry once ported), TP other than 2 or 4, FlyDSL missing, expert / data parallelism or EPLB, `VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4`, a routed-expert backend other than `AITER_MXFP4_BF16`, or fewer than 256 compute units (a partitioned GPU: every CTA of a launch must stay resident). The MoE and shared-expert tensors are checked for the kernels’ shapes and dtypes at the first step, as the attention ones are. A layer outside the kernels’ scope is logged once.
- **All-reduces are exact:** fp32 sum in rank order, then bf16. At ≤ 48 rows vLLM’s own decode all-reduces are below the INT4 quick-reduce threshold, so the current path runs them exact too.
- **A scratch per step width.** Hand-offs between CTAs are tagged mailbox pairs in a scratch laid out per step width. Each width has its own scratch, allocated at its first (eager) step: 3–16 MB, and at most 8 widths under DSpark. So a word one width polls as a tag only ever holds tags. The MoE’s task counters outlive a width and live in the epoch buffer.
- **Dependencies:** the `flydsl` wheel, plus two small AITER helpers that path already depends on: `aiter.ops.flydsl.kernels.buffer_ops` (FlyDSL’s wheel no longer ships it) and `UncachedIpcHeap` for peer memory.

## Duplicate check

- No open PR targets DeepSeek-V4.1 mono decode. Searched “dsv41 mono”, “DeepSeek V4.1 mono”, “mono decode layer” and “dsv41_mono”.
- The sibling mono-decode PRs are for other models: [#59705](https://github.com/aarnphm/aarnphm.github.io/issues/59705) (MiniMax-M3) and [#60216](https://github.com/aarnphm/aarnphm.github.io/issues/60216) (GLM-5).
- This PR hooks per decoder layer rather than replacing the model forward, because only 30 of DSv4.1’s 40 layers are eligible.

## Test Plan

- `pytest tests/models/test_dsv41_mono_decode.py` (CPU, wired into the MI355 Basic Models job): a decode step takes the mono path and hands the kernels `[blocks, block, 584]` cache views. A padded FULL-graph step takes it too (the kernels skip pad rows, whose slot is -1). PIECEWISE steps, prefill or mixed steps, more decode tokens than rows, non-causal windows, missing metadata, more than 48 rows and the first layer all take the original path. Deployments the kernels cannot run raise.
- `pytest tests/models/test_dsv41_mono_numerics.py` (4 × MI355, its own CI step). One layer with random weights in the checkpoint’s formats, at TP2 and TP4: the kernels against the ops vLLM runs for the same decode step (the mHC seams, the ROCm attention chain, the all-reduce, `DeepseekV4MoE`). Each stage is checked on its own input, by cosine and relative error, for the whole layer and the FFN launch, at 1×6 and 8×6 rows. Then both widths’ launches are captured in one graph and replayed 300 times, bit for bit against the first run.
- `pytest tests/models/test_dsv41_mono_layers.py` (same CI step). A five-layer DeepSeek-V4.1-Flash at TP2 and TP4 holding both layer kinds the mono path takes: the whole layer at a ratio-2 and a ratio-1 consumer, and the FFN launch at the sliding-window and KV / index-source layers. It is served with mono on and off on the same random weights, and every decoder layer’s outputs at every decode token must match. It runs `DeepseekV4DecoderLayer.forward` as the engine does, so a change to the layer that the kernels do not follow fails it.
- Race stress: the whole layer and the FFN launch in one HIP graph, 48-row and 6-row steps alternating, every intermediate compared bit for bit with the first run. 100K replays each on layers 3 and 9 (ratio 2) and 21 (ratio 1).
- Compile-only build (`COMPILE_ONLY=1`) of the in-tree kernels: K1 and K2 at TP2 and TP4, M = 6 and 48, both layer types.
- No regression of K1 + K2: K2’s FFN half is factored out and shared with the FFN launch. K2 disassembles instruction-identical to the first commit’s (TP2 M=6 ratio 1, TP4 M=48 ratio 2), and the whole-layer accuracy check reproduces its earlier numbers exactly.
- FFN launch, kernel level: against vLLM’s FFN half (the attention all-reduce, `MHCPreDelayedOp`, `DeepseekV4MoE` with its all-reduce) on the real weights of the 10 layers, with the attention partial from a real ROCm attention chain. 48 points: TP2/TP4 × concurrency 1/2/4/8 × 1–6 tokens per request; the 10 layers chained in one HIP graph, cold weights.
- Kernel level (MI355X): the layer against vLLM’s own modules (`DeepseekV4MoE`, the mHC seams, the ROCm attention chain, the all-reduce) in a TP process group, with real checkpoint weights. 96 points: TP2/TP4 × concurrency 1–8 × 1–6 tokens per request × the two layer types. Timing: 8 layers chained in one HIP graph, cold weights, vLLM’s path captured under `graph_capture`.
- GSM8K: InferenceX’s task, 5-shot, full 1,319, temperature 0, concurrency 8, DSpark block rejection; TP2 for K1 + K2 alone, TP2 and TP4 with the FFN launch.
- AgentX e2e with the [InferenceX#3690](https://github.com/InferenceX/aarnphm.github.io/issues/3690) recipe (this tree’s image, a8w4 MoE, INT4 quick reduce), golden DSpark acceptance 3.51, 3600 s per point, c1/c2/c4/c8 at TP2 and TP4.
- Accuracy, per RFC [#59665](https://github.com/aarnphm/aarnphm.github.io/issues/59665): GPQA-Diamond and AIME25 at TP2, thinking on with `reasoning_effort` high, 8 requests in flight a server (every decode step within 48 rows), against the same build with `VLLM_ROCM_MONO_DECODE=0`. Long context: RULER NIAH (single, multikey 2 and 3) at 32K, 128K and 256K.

## Test Result

**Unit:** 33 passed. **Compile-only:** all K1 / K2 / FFN builds compile.

**Numerics test (GPU):** passed at TP2 and TP4: every token’s output cosine ≥ 0.9993 against vLLM’s chain, relative errors 0.3–0.5% (bounds 1–1.5%; a routed scale off by 3% fails), 300 replays bit-exact. **Race stress:** no mismatch in 100K replays on each layer. Before the per-width scratch, layer 3 diverged at replay 13 on every run. **Model-level test:** passed at TP2 and TP4 (9 of 9 local runs of the final version; 40 of 41 of the TP2-only one before it, the failure not recurring and its log not kept). A routed scale off by 3% fails it at both sizes.

**Accuracy**, TP2, per RFC [#59665](https://github.com/aarnphm/aarnphm.github.io/issues/59665) ([details](https://github.com/vllm-project/vllm/pull/60397#issuecomment-6051909168)):

- pass\@1, against this build with `VLLM_ROCM_MONO_DECODE=0`;
- thinking on, `reasoning_effort` high, temperature 0.6, 8 requests in flight a server;
- ± is the standard error across questions; Δ is paired by question.

|                                     | Reference  | Mono                      |
| ----------------------------------- | ---------- | ------------------------- |
| GPQA-Diamond (8 samples a question) | 85.3 ± 2.2 | 85.1 ± 2.2 (Δ −0.2 ± 0.6) |
| AIME25 (16 samples a problem)       | 99.4 ± 0.3 | 98.3 ± 0.8 (Δ −1.0 ± 0.7) |

RULER NIAH, long context ([details](https://github.com/vllm-project/vllm/pull/60397#issuecomment-6052301592)): single, multikey 2 and multikey 3 at 32K, 128K and 256K, 40 samples a cell. Every cell is 40/40 for both mono and the reference.

**FFN launch, kernel level, µs per layer** (vLLM’s FFN half → the FFN launch, at 6 tokens per request). Every point is faster or at parity: TP2 1.01–2.14×, TP4 1.12–2.32×. The residual and the MoE input match vLLM’s bit for bit at every point; the layer output’s cosine is ≥ 0.99999, and every token’s ≥ 0.9999.

| TP | c1             | c2              | c4              | c8              |
| -- | -------------- | --------------- | --------------- | --------------- |
| 2  | 136→81 (1.69×) | 181→128 (1.41×) | 241→215 (1.12×) | 345→342 (1.01×) |
| 4  | 111→57 (1.96×) | 147→85 (1.74×)  | 176→140 (1.26×) | 234→209 (1.12×) |

**Kernel level, µs per layer.** Every one of 96 points is faster, 1.10–2.18×. Accuracy: residual cosine ≥ 0.99999. MoE output: 1,234 of 1,260 tokens at cosine ≥ 0.999. Each of the other 26 is a top-6 routing near-tie: the reference’s own 6th/7th expert score gap is ≤ 3.2e-3, against medians near 1e-2. On the same FFN input the MoE matches vLLM’s at cosine 0.99999.

| Layers  | TP | Speedup range | DSpark decode points c1 · c2 · c4 · c8 (current → mono µs) |
| ------- | -- | ------------- | ---------------------------------------------------------- |
| ratio 1 | 2  | 1.11–2.04×    | 209→122 · 262→179 · 336→280 · 474→427                      |
| ratio 2 | 2  | 1.10–2.05×    | 210→123 · 255→172 · 325→269 · 446→406                      |
| ratio 1 | 4  | 1.15–2.18×    | 180→94 · 218→123 · 256→192 · 324→282                       |
| ratio 2 | 4  | 1.17–2.18×    | 180→95 · 216→121 · 248→185 · 316→270                       |

**GSM8K** (strict / flexible; DSpark mean acceptance length):

|                                      | TP2                             | TP4                             |
| ------------------------------------ | ------------------------------- | ------------------------------- |
| current path                         | 0.9742 ± 0.0044 / 0.9735; 3.949 | —                               |
| K1 + K2 (30 layers)                  | 0.9750 ± 0.0043 / 0.9742; 3.911 | 0.9719 ± 0.0045 / 0.9712; 4.003 |
| K1 + K2 + the FFN launch (40 layers) | 0.9735 ± 0.0044 / 0.9727; 3.928 | 0.9719 ± 0.0045 / 0.9712; 3.989 |

**AgentX with the FFN launch (all 40 layers) vs K1 + K2 alone (30 layers).** Same recipe and protocol. AgentX replays a fixed request sequence, so the runs’ requests match one to one; the last column is the median ratio of a request’s inter-token latency (FFN launch / K1 + K2).

| TP | conc | p90 interactivity, tok/s/user | total throughput, tok/s | decode, same requests |
| -- | ---- | ----------------------------- | ----------------------- | --------------------- |
| 4  | c1   | 523.7 → 571.4 (1.09×)         | 26,796 → 28,225 (1.05×) | 0.915                 |
| 4  | c2   | 471.1 → 510.0 (1.08×)         | 27,217 → 28,241 (1.04×) | 0.916                 |
| 4  | c4   | 445.1 → 502.5 (1.13×)         | 36,791 → 39,640 (1.08×) | 0.911                 |
| 4  | c8   | 358.6 → 367.6 (1.02×)         | 62,621 → 62,877 (1.00×) | 0.915                 |
| 2  | c1   | 429.1 → 471.5 (1.10×)         | 25,039 → 25,599 (1.02×) | 0.928                 |
| 2  | c2   | 381.7 → 417.2 (1.09×)         | 25,695 → 25,962 (1.01×) | 0.925                 |
| 2  | c4   | 332.4 → 394.4 (1.19×)         | 35,885 → 36,761 (1.02×) | 0.927                 |
| 2  | c8   | 245.3 → 265.7 (1.08×)         | 60,501 → 61,180 (1.01×) | 0.932                 |

Decode on the same requests is 7–9% faster at every point. TP2 is a re-run: the first TP2 round hit node-wide engine stalls of 20–40 s, on all four servers at the same moments, that cost its throughput. Against InferenceX’s run, the FFN launch reaches 1.56× / 1.46× / 1.47× / 1.30× p90 interactivity and 1.15× / 1.14× / 1.11× / 1.05× total throughput at TP2 c1 / c2 / c4 / c8, and 1.66× / 1.56× / 1.64× / 1.38× and 1.19× / 1.20× / 1.14× / 1.04× at TP4. The tables below are K1 + K2 alone.

**AgentX, TP2.** Against the same-node mono-off baseline, run the same way (four points at once on one node):

| conc | p90 interactivity, tok/s/user | total throughput, tok/s | TTFT p90, s |
| ---- | ----------------------------- | ----------------------- | ----------- |
| c1   | 318.1 → 429.1 (1.35×)         | 21,573 → 25,039 (1.16×) | 1.18 → 1.28 |
| c2   | 299.3 → 381.7 (1.28×)         | 22,735 → 25,695 (1.13×) | 1.10 → 1.08 |
| c4   | 288.0 → 332.4 (1.15×)         | 33,043 → 35,885 (1.09×) | 1.27 → 1.26 |
| c8   | 214.6 → 245.3 (1.14×)         | 58,294 → 60,501 (1.04×) | 1.01 → 1.08 |

**AgentX, TP4.** Against InferenceX’s own run of the same image and recipe ([run 37510490434](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/37510490434)). The TP2 same-node baseline above matches InferenceX’s TP2 run within a few percent, so InferenceX serves as the TP4 reference.

| conc | p90 interactivity, tok/s/user | total throughput, tok/s |
| ---- | ----------------------------- | ----------------------- |
| c1   | 343.7 → 523.7 (1.52×)         | 23,691 → 26,796 (1.13×) |
| c2   | 327.4 → 471.1 (1.44×)         | 23,579 → 27,217 (1.15×) |
| c4   | 307.2 → 445.1 (1.45×)         | 34,831 → 36,791 (1.06×) |
| c8   | 265.8 → 358.6 (1.35×)         | 60,266 → 62,621 (1.04×) |

**Provenance of the numbers.** The kernel, GSM8K and AgentX numbers come from these kernels while they were hosted in AITER during development ([ROCm/aiter#6196](https://github.com/ROCm/aiter/issues/6196)). Moving them here changed only import paths, the merge of a few small helper modules, and formatting. The kernel-level accuracy check re-run against this in-tree copy (TP2 layer 21 M=48, TP2 layer 3 M=6, TP4 layer 21 M=48) reproduces the earlier results exactly, to six decimals of cosine. The TP4 GSM8K and every FFN-launch number come from the in-tree code. The kernel, GSM8K and AgentX numbers predate the per-width scratch. It leaves kernel time unchanged: 24 TP2/TP4 points, mean ratio 1.000, range 0.996–1.009.

## AI assistance

This change was written with AI assistance (Claude Code). It is opened as a draft for the submitter to review every changed line before it is marked ready.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**Fangzhou-Ai** (2026-10-07T21:13:26Z):

/ci run

**github-actions** (2026-10-07T21:14:24Z):

✅ Triggered [Buildkite CI #93417](https://buildkite.com/vllm/ci/builds/93417) for commit `d18f7697a6bc`.

**benchislett** (2026-10-07T21:16:51Z):

Given the scope of this change, I don’t think GSM8k only is sufficient for correctness testing. Can you please ensure that at least GPQA-D and a hard eval like AIME25 are run to ensure there is no numerical leakage?

I haven’t formalized this proposal yet but I outline some eval guidance in this RFC, [which may be helpful](https://github.com/vllm-project/vllm/issues/59665).

**Fangzhou-Ai** (2026-10-08T00:23:03Z):

/ci run

**github-actions** (2026-10-08T00:23:23Z):

✅ Triggered [Buildkite CI #93467](https://buildkite.com/vllm/ci/builds/93467) for commit `524bc76eeb3e`.

**tjtanaa** (2026-10-08T00:59:04Z):

> Given the scope of this change, I don’t think GSM8k only is sufficient for correctness testing. Can you please ensure that at least GPQA-D and a hard eval like AIME25 are run to ensure there is no numerical leakage?
>
> I haven’t formalized this proposal yet but I outline some eval guidance in this RFC, [which may be helpful](https://github.com/vllm-project/vllm/issues/59665).

To add on, can we have long context evaluation as well? e.g. NIAH

**vllm-agent** (2026-10-08T01:07:43Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 84 test steps (150 jobs) instead of 57 (90 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector  | Would skip | Would add |
| ---------------------- | ------------- | --------- | ---------- | --------- |
| NVIDIA, CPU and others | 57 (90)       | 84 (150)  | 15 (15)    | 42 (75)   |
| AMD mirrors            | 55 (97)       | 162 (226) | 0 (0)      | 107 (129) |

<details><summary>Selector would run (84)</summary>

- `amd-fp8-moe-kernels-mi355`
- `amd-kernels-mi355`
- `amd-lm-eval-small-models-harness`
- `amd-native-quantization-kernels-mi355`
- `amd-qwen3-next-mtp-async-eplb-accuracy`
- `arm-cpu-test` ×3
- `async-engine-inputs-utils-worker`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `basic-models-tests-extra-initialization` ×14
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `batch-invariance-b200`
- `batch-invariance-h100`
- `cpu-kernel-tests` ×2
- `cpu-language-generation-and-pooling-model-tests` ×3
- `cpu-multi-modal-model-tests-n` ×4
- `cpu-multimodal-config`
- `cpu-params-env-tokenizers-parser`
- `cpu-reasoning-renderers`
- `cpu-tool-parsers`
- `cudagraph`
- `deepseek-v4-kernel-test-b200`
- `deepseek-v4-kernel-test-h100`
- `distributed-compile-comm-4-gpus`
- `distributed-compile-unit-tests-2xh100`
- `distributed-dp-tests-4-gpus`
- `distributed-model-tests-2-gpus` ×3
- `distributed-torchrun-examples-4-gpus`
- `e2e-core-1-gpu`
- `e2e-core-large-memory`
- `engine`
- `engine-1-gpu`
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `entrypoints-unit-tests`
- `fusion-and-compile-unit-tests-2xb200`
- `fusion-e2e-tp2-ar-rms-dynamo-partition-amd`
- `fusion-e2e-tp2-ar-rms-inductor-partition-amd`
- `glm5next-unit-tests`
- `inkling-unit-tests-b200`
- `kernels-attention-test` ×7
- `kernels-b200` ×3
- `kernels-core-operation-test` ×3
- `kernels-flashmla-test-h100`
- `kernels-mamba-test`
- `kernels-mhc-test-b200`
- `kernels-moe-test` ×5
- `kernels-quantization-test` ×6
- `kernels-root-misc-test-b200`
- `kimi-k3-prefix-cache-4xb200` ×2
- `kimi-k3-unit-tests-b200`
- `language-models-tests-extra-standard` ×2
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `model-executor`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `pipeline-context-parallelism-4-gpus`
- `pytorch-compilation-passes-unit-tests`
- `pytorch-compilation-unit-tests`
- `pytorch-compilation-unit-tests-h100`
- `quantization` ×4
- `qwen4-exp-unit-tests`
- `qwen4-exp-unit-tests-cpu`
- `rayexecutorv2-4-gpus`
- `replayssm-e2e`
- `rust-frontend-openai-coverage`
- `rust-frontend-serve-admin-coverage`
- `spec-decode-draft-model` ×4
- `v1-attention-b200` ×2
- `v1-attention-h100-mi300` ×2
- `v1-core`
- `v1-kv-connectors` ×4
- `v1-others-cpu`

</details>

<details><summary>Would skip (today's rules run them) (15)</summary>

- `ascend-npu-test`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `benchmarks-cli-test`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `kernels-fla-ops-test-b200`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `platform-tests`
- `pytorch-compilation-dynamic-shapes`
- `pytorch-fullgraph-cudagraph-l4-compatibility`
- `pytorch-fullgraph-test`
- `rl-entrypoints-tests`

</details>

<details><summary>Would add (today's rules do not run them) (42)</summary>

- `amd-kernels-mi355` (code map)
- `amd-lm-eval-small-models-harness` (code map)
- `amd-qwen3-next-mtp-async-eplb-accuracy` (code map)
- `arm-cpu-test` ×3 (code map)
- `async-engine-inputs-utils-worker` (code map)
- `batch-invariance-b200` (code map)
- `batch-invariance-h100` (code map)
- `cpu-kernel-tests` ×2 (code map)
- `cpu-multi-modal-model-tests-n` ×4 (code map)
- `cudagraph` (code map)
- `deepseek-v4-kernel-test-b200` (code map)
- `deepseek-v4-kernel-test-h100` (code map)
- `distributed-compile-comm-4-gpus` (code map)
- `distributed-compile-unit-tests-2xh100` (code map)
- `distributed-torchrun-examples-4-gpus` (code map)
- `e2e-core-1-gpu` (code map)
- `e2e-core-large-memory` (code map)
- `engine-1-gpu` (code map)
- `entrypoints-unit-tests` (code map)
- `fusion-and-compile-unit-tests-2xb200` (code map)
- `fusion-e2e-tp2-ar-rms-dynamo-partition-amd` (code map)
- `fusion-e2e-tp2-ar-rms-inductor-partition-amd` (code map)
- `kernels-attention-test` ×7 (code map)
- `kernels-b200` ×3 (code map)
- `kernels-core-operation-test` ×3 (code map)
- `kernels-flashmla-test-h100` (code map)
- `kernels-mamba-test` (code map)
- `kernels-quantization-test` ×6 (code map)
- `language-models-tests-extra-standard` ×2 (code map)
- `model-executor` (code map)
- `pipeline-context-parallelism-4-gpus` (code map)
- `quantization` ×4 (code map)
- `rayexecutorv2-4-gpus` (code map)
- `replayssm-e2e` (code map)
- `rust-frontend-openai-coverage` (code map)
- `rust-frontend-serve-admin-coverage` (code map)
- `spec-decode-draft-model` ×4 (code map)
- `v1-attention-b200` ×2 (code map)
- `v1-attention-h100-mi300` ×2 (code map)
- `v1-core` (code map)
- `v1-kv-connectors` ×4 (code map)
- `v1-others-cpu` (code map)

</details>

<details><summary>AMD mirrors: would skip (0)</summary>

none

</details>

<details><summary>AMD mirrors: would add (107)</summary>

- `async-engine-inputs-utils-worker` (code map)
- `batch-invariance-h100` (code map)
- `crosslayer-kv-layout-distributed-nixlconnector-pd-accuracy-tests-4-gpus` (code map)
- `cudagraph` (code map)
- `deepseek-v4-kernel-test-h100` (code map)
- `distributed-comm-ops` (code map)
- `distributed-compile-comm-4-gpus` (code map)
- `distributed-compile-rpc-tests-2-gpus` (code map)
- `distributed-compile-unit-tests-2xh100` (code map)
- `distributed-dp-tests-2-gpus` (code map)
- `distributed-flashinfer-nixlconnector-pd-accuracy-4-gpus` (code map)
- `distributed-mooncakeconnector-pd-accuracy-4-gpus` (code map)
- `distributed-nixlconnector-pd-accuracy-4-gpus` (code map)
- `distributed-tests-8xh100` (code map)
- `distributed-torchrun-examples-4-gpus` (code map)
- `distributed-torchrun-shutdown-tests-2-gpus` (code map)
- `docker-build-metadata` (code map)
- `dp-ep-distributed-nixlconnector-pd-accuracy-tests-4-gpus` (code map)
- `e2e-core-1-gpu` (code map)
- `e2e-core-large-memory` (code map)
- `e2e-scheduling-1-gpu` (code map)
- `e2e-scheduling-accuracy-1-gpu` (code map)
- `engine-1-gpu` (code map)
- `entrypoints-unit-tests` (code map)
- `eplb-algorithm` (code map)
- `eplb-execution` (code map)
- `examples` (code map)
- `extract-hidden-states-integration` (code map)
- `extract-hidden-states-integration-2-gpus` (code map)
- `fault-tolerance-e2e-2xh100` (code map)
- `fusion-and-compile-unit-tests-2xb200` (code map)
- `fusion-e2e-config-sweep-h100` (code map)
- `fusion-e2e-quick-h100` (code map)
- `fusion-e2e-tp2-ar-rms-config-sweep-h100` (code map)
- `fusion-e2e-tp2-asynctp-config-sweep-h100` (code map)
- `fusion-e2e-tp2-b200` (code map)
- `fusion-e2e-tp2-quick-h100` (code map)
- `gemm-rs-ar-2xb200` (code map)
- `hybrid-ssm-nixlconnector-pd-accuracy-tests-4-gpus` (code map)
- `hybrid-ssm-nixlconnector-pd-prefix-cache-2-gpus` (code map)
- `kernels-attention-diffkv-test-h100` (code map)
- `kernels-core-operation-test` ×3 (code map)
- `kernels-flashmla-test-h100` (code map)
- `kernels-fusedmoe-layer-test-2-b200s` (code map)
- `kernels-fusedmoe-layer-test-2-h100s` (code map)
- `kernels-helion-test` ×5 (code map)
- `kernels-mamba-test` (code map)
- `kv-offload-large` (code map)
- `kv-offload-medium` (code map)
- `kv-offload-small` (code map)
- `language-models-tests-extra-standard` ×2 (code map)
- `lm-eval-dspark-watermark-2xh100` (code map)
- `lm-eval-small-models` (code map)
- `lm-eval-turboquant-k3v4nc` (code map)
- `lm-eval-turboquant-k8v4` (code map)
- `lm-eval-turboquant-t3nc` (code map)
- `lm-eval-turboquant-t4nc` (code map)
- `lm-eval-watermarking` (code map)
- `lora` ×4 (code map)
- `lora-tp-distributed` ×4 (code map)
- `metrics-tracing-2-gpus` (code map)
- `model-executor` (code map)
- `mooncake-ec-tcp-e2e-2-gpus` (code map)
- `mrcr-eval-small-models` (code map)
- `multi-modal-accuracy-eval-small-models` (code map)
- `multiconnector-nixl-offloading-pd-accuracy-2-gpus` (code map)
- `multiconnector-nixl-offloading-pd-edge-cases-2-gpus` (code map)
- `nixlconnector-pd-edge-cases-2-gpus` (code map)
- `nixlconnector-pd-spec-decode-acceptance-2-gpus` (code map)
- `openai-api-correctness` (code map)
- `pipeline-context-parallelism-4-gpus` (code map)
- `plugin-tests-2-gpus` (code map)
- `push-nixlconnector-pp-prefill-pd-accuracy-4-gpus` (code map)
- `pytorch-nightly-dependency-override-check` (code map)
- `quantization` ×4 (code map)
- `quantized-fusions` (code map)
- `quantized-models-test` (code map)
- `ray-dependency-compatibility-check` (code map)
- `rayexecutorv2-4-gpus` (code map)
- `regression` (code map)
- `rust-frontend-core-correctness` (code map)
- `rust-frontend-distributed` (code map)
- `rust-frontend-openai-coverage` (code map)
- `rust-frontend-serve-admin-coverage` (code map)
- `rust-frontend-tool-use` (code map)
- `samplers-multimodal-beam-search` (code map)
- `samplers-test` (code map)
- `scale-out-ec-e2e-2-gpus` (code map)
- `sharded-rdt-weight-transfer` (code map)
- `spec-decode-draft-model` ×4 (code map)
- `spec-decode-eagle-1-deepseek-qwen` (code map)
- `spec-decode-eagle-2-llama3-qwen-vl-other` (code map)
- `spec-decode-mtp-deepseek-mimo` (code map)
- `spec-decode-mtp-gemma4` (code map)
- `spec-decode-mtp-qwen3-5` (code map)
- `spec-decode-ngram-suffix` (code map)
- `spec-decode-speculators` (code map)
- `torch-stable-abi-audit` (code map)
- `v1-core` (code map)
- `v1-executor-worker` (code map)
- `v1-kv-connectors` ×4 (code map)
- `v1-kv-offload` (code map)
- `v1-logits-oracle` (code map)
- `v1-metrics-lmeval` (code map)
- `v1-sample` (code map)
- `v1-spec-decode` (code map)
- `vllm-ir-tests` (code map)

</details>

<sub>32 changed files · base `1fbbfa1657` · head `ba010ab89e` · Python record: build 93039 at `1e5d0ea888` · kernel record: table [`43b4aae`](https://github.com/aarnphm/aarnphm.github.io/commit/43b4aaea3e) (build 93244), map [`43b4aae`](https://github.com/aarnphm/aarnphm.github.io/commit/43b4aaea3e) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 77 optional steps the selector would also run</sub>

**Fangzhou-Ai** (2026-10-08T04:00:31Z):

@benchislett @tjtanaa Thanks, this was worth running: chasing the first round’s AIME25 gap turned up a real bug, now fixed in [`886a5f5`](https://github.com/aarnphm/aarnphm.github.io/commit/886a5f5fed). The evals follow the RFC’s guidance:

- GPQA-Diamond and AIME25, pass\@1 over ≥ 8 samples a question;
- thinking on, with `reasoning_effort` high;
- a reference run of the same build;
- the deployment scenario reproduced: 8 requests in flight a server with 5 DSpark drafts, so every decode step is within the kernels’ 48 rows and takes the mono path.

**Setup.** MI355X, DeepSeek-V4.1-Flash, TP2, the InferenceX MI355X recipe with DSpark (real block rejection).

- Reference: this build with `VLLM_ROCM_MONO_DECODE=0`, which runs main’s path.
- Sampling: temperature 0.6, top-p 0.95.
- GPQA-Diamond: InferenceX’s lm-eval task, each question in 2 option orders, 32K tokens, 4 seeds (8 samples a question).
- AIME25: lm-eval’s `aime25` with the boxed-answer instruction, 64K tokens, 16 seeds.
- Statistics: ± is the standard error across questions; Δ is mono − reference, paired by question.

| TP2, pass\@1 | Reference (mono off) | Mono, before the fix      | Mono, with the fix        |
| ------------ | -------------------- | ------------------------- | ------------------------- |
| GPQA-Diamond | 85.3 ± 2.2           | 84.8 ± 2.3 (Δ −0.5 ± 0.6) | 85.1 ± 2.2 (Δ −0.2 ± 0.6) |
| AIME25       | 99.4 ± 0.3           | 97.9 ± 1.1 (Δ −1.5 ± 0.9) | 98.3 ± 0.8 (Δ −1.0 ± 0.7) |

With the fix, AIME25 is within noise of the reference. Its two remaining empty samples are q4 seed 7 and q13 seed 4. q13 is a long problem: about 40K tokens at the median, up to about 61K. q4 normally takes about 3K tokens, so a loop there is a clear derailment. To check whether mono derails more often, q4 was sampled 96 more times on each arm, with the same settings and 8 in flight. Mono answered 96/96 with no loop. The reference answered 95/96: one sample looped to the 64K cap. Across all q4 samples, the reference loops as often as mono (1 in 120 vs 1 in 112), so these loops come from the model at this temperature, not from the kernels.

**The bug.** Before the fix, mono was 1.5 pp below the reference on AIME25. Four of its 480 samples ran out the 64K budget without an answer; the reference had none. Looking for the cause, a race stress test found one. The test replays the whole layer and the FFN launch in one CUDA graph, alternating 48-row and 6-row steps, and compares every intermediate bit for bit with the first run.

The race: the kernels hand data between CTAs through tagged mailbox pairs, in a scratch laid out per step width. Every width shared one buffer, so a word one width polls as a tag could hold another width’s plain data. In the failing case, 6-row attention partials lay under 48-row tag words. When one equaled the current tag, a poll took a stale value as published. It came from this PR’s adaptation of ATOM’s kernels, not from ATOM. ATOM zeroes every mailbox buffer before each step, which also wipes whatever another width left. This PR replaced those clears with epoch tags, to drop the host step hook and stay graph-safe, but kept the shared scratch. The fix gives each width its own scratch. The stress test is now clean over 100K replays on each of three layers, and kernel time is unchanged (24 points, mean ratio 1.000).

**Numerical closeness, sampling noise removed** (measured before the fix). Greedy decoding of the same 128 prompts (30 AIME25 + 98 GPQA-D, 4,096 tokens), with top-5 logprobs, on four TP2 servers: mono on, mono off, and a second of each as controls. Until two runs diverge they see identical inputs, so their logprobs compare directly:

| Pair                   | Matched prefix (median) | \|Δlogprob\| on it: mean / p99 / max | Top-1 − top-2 margin where they diverge: median / p90 / max |
| ---------------------- | ----------------------- | ------------------------------------ | ----------------------------------------------------------- |
| mono vs reference      | 20 tokens               | 0.030 / 0.31 / 0.58                  | 0.25 / 0.75 / 1.75                                          |
| reference vs reference | 20 tokens               | 0.029 / 0.29 / 0.54                  | 0.25 / 0.75 / 1.50                                          |
| mono vs mono           | 21 tokens               | 0.029 / 0.31 / 0.67                  | 0.25 / 0.75 / 2.25                                          |

The reference diverges from itself as fast as mono diverges from it, every pair at near-ties of bf16 logits, with the same per-token logprob differences.

Also measured before the fix:

- TP4 mono: GPQA-Diamond 85.2 ± 2.2 (4 samples a question) and AIME25 100.0 (240/240).
- GSM8K (5-shot, full 1,319): mono 0.9735 at TP2 and 0.9719 at TP4, against 0.9742 for the reference.
- The per-layer comparison against vLLM’s modules, in the PR description.

A GPU numerics test of the kernels against vLLM’s decode ops now runs in CI ([`e69add8`](https://github.com/aarnphm/aarnphm.github.io/commit/e69add8dfc)).

**tjtanaa** (2026-10-08T04:08:54Z):

Please run NIAH evaluation as well.

**Fangzhou-Ai** (2026-10-08T04:33:12Z):

@tjtanaa Both done.

**Numerics test.** `tests/models/test_dsv41_mono_numerics.py` ([`e69add8`](https://github.com/aarnphm/aarnphm.github.io/commit/e69add8dfc)) runs as its own MI355 CI step: 2 GPUs, about 30 s plus the kernel builds.

It builds one DSv4.1 layer at TP2 with random weights in the checkpoint’s formats: MXFP8 attention linears, MXFP4 routed experts, an MXFP8 shared expert. Replicated weights are the same on both ranks. It then runs the mono kernels against the ops vLLM itself runs for the same decode step:

- the mHC seams (`MHCPreDelayedOp`);
- the ROCm attention chain (`rocm_mxfp8_block32_gemm`, `fused_q_kv_rmsnorm`, `fused_deepseek_v4_qnorm_rope_kv_rope_quant_insert`, `triton_sparse_mla_fwd`, the inverse-RoPE requant, `wo_a`, `wo_b`);
- the all-reduce;
- `DeepseekV4MoE`.

Each stage is checked on its own input, so a failure names the stage:

- the attention seam’s outputs: the residual and the post / comb / pre mixes;
- the residual and the FFN input after the FFN seam;
- the MoE, on the kernels’ own FFN input;
- the layer’s output, per token;
- the FFN launch, from the same unreduced `wo_b` output.

Cases: a 1×6 step on a ratio-1 layer and an 8×6 step on a ratio-2 layer, which are the DSpark shapes. Both run over paged SWA and compressed caches written the way vLLM’s insert writes them, with 2K–40K-token contexts. Any change under `vllm/models/deepseek_v41/amd/` triggers it.

**Long context: RULER NIAH.** Setup:

- TP2, mono on vs off, the same build and recipe as the GPQA / AIME runs.
- Thinking on, greedy, 8 requests in flight a server.
- 40 samples a cell, from lm-eval’s RULER tasks.

| Correct of 40, mono / reference | 32K     | 128K    | 256K    |
| ------------------------------- | ------- | ------- | ------- |
| `niah_single_1`                 | 40 / 40 | 40 / 40 | 40 / 40 |
| `niah_multikey_2`               | 40 / 40 | 40 / 40 | 40 / 40 |
| `niah_multikey_3`               | 40 / 40 | 40 / 40 | 40 / 40 |

Every cell is 100% on both arms. The 256K prompts are about 261K tokens, and every answer contains its needles. These runs, like the post-fix GPQA-D / AIME25 above, include the per-width scratch fix ([`886a5f5`](https://github.com/aarnphm/aarnphm.github.io/commit/886a5f5fed)).

**Fangzhou-Ai** (2026-10-08T04:33:51Z):

/ci run

**github-actions** (2026-10-08T04:34:09Z):

✅ Triggered [Buildkite CI #93507](https://buildkite.com/vllm/ci/builds/93507) for commit `ba010ab89ecc`.

**Fangzhou-Ai** (2026-10-08T16:54:51Z):

/ci run

**github-actions** (2026-10-08T16:55:36Z):

✅ Triggered [Buildkite CI #93658](https://buildkite.com/vllm/ci/builds/93658) for commit `73f40edda6ec`.

**Fangzhou-Ai** (2026-10-09T09:00:25Z):

@mawong-amd Thanks for the thorough review and the independent GSM8K run. Addressed in [`dea9fa3`](https://github.com/aarnphm/aarnphm.github.io/commit/dea9fa3b95) and [`b9423a8`](https://github.com/aarnphm/aarnphm.github.io/commit/b9423a8c7d):

**1. Unsupported deployments now raise at model init.** With `VLLM_ROCM_MONO_DECODE=1`, these raise instead of reading garbage or hanging:

- expert parallelism, EPLB or data parallelism;
- `VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4`;
- a routed-expert backend other than `AITER_MXFP4_BF16`.

Checked e2e: your recipe plus `--enable-expert-parallel` now stops at startup with `VLLM_ROCM_MONO_DECODE needs tensor parallelism only: no expert or data parallelism.`. Per-row 1x32 shared-expert scales are caught by the tensor check in (3).

**2. Relative-error bounds.** Every cosine check in the numerics test now has a relative-L2 bound beside it, at about 3x what the kernels show: 1% on the residual, 1.2% on the FFN input, 1.5% on the MoE and the FFN launch (the kernels measure 0.3–0.5%). Your negative control, `SCALE` 1.5 → 1.45, now fails.

**3. MoE tensors are checked.** `MonoLayerWeights.check` covers the router, all four routed-expert tensors and the shared expert’s, with shapes derived from the TP size (including the padded intermediate). It runs at the first step, like `AttnWeights.check`. The quant-method class check is the backend check in (1).

**4. Liveness.** `create` now requires at least 256 compute units, so a CPX/DPX partition raises instead of deadlocking. Bounded spins with `s_trap` are a fair point, but they put a counter and a compare in every poll loop on the hot path. I’d rather do that as a follow-up with a perf A/B than fold it in here.

**5. In-tree stress.** The numerics test now captures both step widths’ launches (1×6 and 8×6 rows, whole layer and FFN launch) in one graph and replays it 300 times, bit for bit against the first run. One caveat: the race this PR fixed needed real-weight data to equal a tag at particular epochs. A few hundred random-weight replays guard the hand-off protocol in general rather than that exact collision. The 100K-replay real-weight version stays a development tool.

**6. Wiring.** Agreed on all four; I’d take them as follow-ups:

- building the runner at load (it needs a TP collective there);
- one decision per step;
- dropping the cached weights on reload;
- one home for `common/`, shared with [#59705](https://github.com/aarnphm/aarnphm.github.io/issues/59705), [#60683](https://github.com/aarnphm/aarnphm.github.io/issues/60683) and [#60793](https://github.com/aarnphm/aarnphm.github.io/issues/60793), which is better settled with those authors than in this PR.

**Nits**

- **TP4 coverage:** the numerics test runs TP2 and TP4, and its CI step moved to `mi355_4`. Running TP4 found that the random routed weights filled the loader’s 576 → 640 padding; they now keep it zero, as the loader does.
- **`rocm.py` `reduce_results`:** the description now notes the main-branch fix.
- **Padded batches:** right, the description was wrong. Padded FULL-graph steps take the kernels, and pad rows are skipped via `slot < 0`. Corrected, with a CPU test for it.
- **Top-k count:** the valid entries are always a prefix of exactly min(n, 512). At decode the indexer’s top-k is `top_k_per_row_decode` over each row’s `[0, n)`, which pads rows shorter than k with -1 after the valid entries. Candidate masking (`candidate_topk_blocks` 2048 × `candidate_block_size` 8 = 16,384 positions) leaves at least min(n, 16384) ≥ min(n, 512) positions unmasked. So every row has min(n, 512) valid entries first, which is the count `front.py` computes.
- **`x8` / `x8s`:** `scratch_layout` now asserts that region names are unique. The one deliberate exception is this pair: K1’s and the MoE’s MXFP8 x are both plain words of the same size, so they share the MoE’s region. Behavior and layout are unchanged.

Tests: CPU 33/33. Numerics at TP2 and TP4 pass on MI355X. A TP2 server with the recipe starts and serves with mono on.

**Fangzhou-Ai** (2026-10-09T09:10:57Z):

/ci run

**github-actions** (2026-10-09T09:11:18Z):

✅ Triggered [Buildkite CI #93808](https://buildkite.com/vllm/ci/builds/93808) for commit `9a24c32a1743`.

**WoosukKwon** (2026-10-09T20:28:51Z):

Excited to learn more about the kernel! 🚀

**Fangzhou-Ai** (2026-10-09T22:11:19Z):

/ci run

**github-actions** (2026-10-09T22:11:38Z):

✅ Triggered [Buildkite CI #93923](https://buildkite.com/vllm/ci/builds/93923) for commit `03705674ac4c`.

**Fangzhou-Ai** (2026-10-09T23:20:25Z):

/ci run

**github-actions** (2026-10-09T23:20:44Z):

❌ This PR is 4 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:6090922901 -->

**Fangzhou-Ai** (2026-10-09T23:21:22Z):

/ci run

**github-actions** (2026-10-09T23:21:42Z):

✅ Triggered [Buildkite CI #93936](https://buildkite.com/vllm/ci/builds/93936) for commit `f6b54daade30`.

