---
title: '60904'
pageLayout: default
slug: thoughts/inference/vllm-prs/60904
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/60904.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#60904](https://github.com/aarnphm/aarnphm.github.io/issues/60904) \[RFC\]: A standard for Specialized (MonoKernels) Decodes in vllm/models

[vllm-project/vllm#60904](https://github.com/vllm-project/vllm/issues/60904) · dllehr-amd · OPEN · created 2026-10-09T18:25:05Z

### Motivation.

A **Specialized Decode** implementation is one where most/all operations in a model’s forward pass are manually implemented in a way that optimizes for the model’s specific shapes and execution patterns on certain hardware. Typically, these include heavy kernel fusions and overlap for low-latency execution, and custom kernels that tune exact grid sizes for certain batch sizes. These implementations can reach tens of thousands of lines of code, and present a major challenge in managing their complexity. This RFC proposes a standard way to modularize and integrate these runtimes into vLLM under `vllm/models/<model>/<vendor>/`.

#### Why Specialized Decodes like MonoKernels?

At low concurrency, decode is bound by launch overhead and weight reads, and a MonoKernel removes most of both. On 8× MI350X a graph-mode GLM-5.2 decode step issues \~2,459 kernels in vLLM. With the MonoKernel it issues \~210: 75 MonoKernel launches plus \~135 for dense layers 0–2, input prep and the head.

| GLM-5.2-MXFP4 ([#60889](https://github.com/aarnphm/aarnphm.github.io/issues/60889)), TP8, MI350X, \~3k-token context | vLLM                      | MonoKernel      | Speedup                              |
| -------------------------------------------------------------------------------------------------------------------- | ------------------------- | --------------- | ------------------------------------ |
| Time per token, batch 1 / 8, defaults (BF16 attention)                                                               | 13.70 / 17.33 ms          | 5.92 / 11.44 ms | 2.31× / 1.51×                        |
| Time per token, batch 1, FP8 attention opt-in                                                                        | 13.70 ms                  | 4.80 ms         | 2.85×                                |
| MoE layer w/o indexer (57 of 75), batch 4 / 8, FP8                                                                   | 178.8 / 189.6 µs          | 69.1 / 100.6 µs | 2.59× / 1.88×                        |
| MoE layer with indexer (18 of 75), batch 4 / 8, FP8                                                                  | 214.9 / 227.6 µs          | 88.5 / 123.4 µs | 2.43× / 1.84×                        |
| AgentX time between tokens, 1 / 8 users (3,600 s/point), FP8                                                         | 15.0 / 20.4 ms            | 6.5 / 12.7 ms   | 2.28× / 1.68×                        |
| GSM8K strict, 1,319 questions, graphs, BF16                                                                          | 1228, 1234, 1235 (3 runs) | 1240            | parity: paired McNemar p = 0.22–0.63 |

Layer times are profiler-corrected, from one run. The gain shrinks with batch size, so the target is latency-bound serving: agents, chat and small-batch decode.

### Proposed Change.

#### Goals

1. **Opt-in:** off by default, enabled with one switch (`additional_config={"decode_backend": "latency_specialized"}` today).
2. **Refuse at startup:** an unsupported deployment fails with one error listing every reason, never a silent fallback.
3. **The model owns it:** code lives under `vllm/models/<model>/<vendor>/specialized_decode/`, with no core changes.
4. **One `plan()` per step:** it picks the specialized kernels or vLLM’s stock forward; prefill and mixed batches always run stock.
5. **One `step()` or `run()` per pass** If specialized_decode exists, run the `step()` or `run()` function, else call stock
6. **Fixed batch sizes:** the decode kernel declares the batch sizes it serves; larger batches run stock.
7. **Avoid duplicating weights** where prudent, reading vLLM’s loaded weights and caches in place.

#### Layout

Layout (shown for a monokernel; Kimi-K3 uses `nvidia/` and `cutedsl/`):



```
vllm/models/<model>/<vendor>/
    model.py                 # calls specialized_decode_config() in __init__, plan() in forward
    model_state.py           # MRV2 ModelState: builds the decode after load, binds KV caches
    specialized_decode/
        README.md            # supported checkpoints, refusals, options
        runtime/
            __init__.py      # DecodeConfig, batch sizes, specialized_decode_config(), unsupported()
            decode.py        # SpecializedDecode: plan(), run() or Step.layer()
            ...              # weight views, guards, model-specific helpers
        mono/                # the kernels, when the decode uses monokernels
tests/models/<model>/        # proposed: CPU tests for refusals and plan() routing
```

Instantiation of specialized decode happens at startup



```
self.specialized_decode_config = specialized_decode_config(vllm_config)
self.specialized_decode = SpecializedDecode(
            self.model, self.specialized_decode_config
        )
```

Two methods to Available.

Replace the whole forward function with the specialized decode (and let it decide when to call the stock version)



```python
# Kimi-K3 (step mode, #61867): the whole step runs in the specialized decode
decode = self.specialized_decode
step = decode.plan(positions, intermediate_tensors)
if step is not None:
  return decode.run(step, input_ids, inputs_embeds)
return self.model(input_ids, positions, intermediate_tensors, inputs_embeds)
```

Replace the Decode Layer within the forward function with the specific kernel to run, and let the forward function decide.



```python
# GLM-5.2 (layer mode, #60889): a kernel per layer, or vLLM's layer
step = None if decode is None else decode.plan(positions, intermediate_tensors)
...
kernel = step.layer(idx) if step is not None else None
hidden_states, residual = (kernel or layer)(positions, hidden_states, residual)
```

**Not in Scope**

- Prefill and mixed steps. These always run vLLM’s layers.
- Models compiled with `@support_torch_compile`.
- Launches that span more than two layer (some fusions may fuse end of one layer with beginning of another)
- No Runtime fallback. If a megakernel fails the engine should stop.

##### Enabling and refusal

- **Switch:** `--monokernel` (with `--no-monokernel`), the same for every model and off by default, like `--async-scheduling`. It sets `KernelConfig.monokernel`, so it also works from a config file or the Python API.
  - Additionally we can use `--additional_config={"decode_backend": "latency_specialized"}` As this will be used in the more generic case.
  - It applies to the loaded model’s spec. A model has at most one spec per vendor, so there is no name to pick.
  - There are no other user knobs. Tuning lives in the spec’s `options`.
- **Refusal** happens in three stages: config (before load), model (after load) and caches (at every bind). Each stage reports every reason, not only the first. If the switch is on and the config is refused, startup raises.
- **Default-deny:** any feature in use that `supports` doesn’t list is refused, then the spec’s own checks run. The features are: PP, DP, EP, EPLB, DCP, PCP, LoRA, KV connectors, routed-expert capture, speculative decoding, sleep mode, CPU weight offload, weight updates, partitioned GPUs, and EAGLE3 aux layers among the kernel layers.
- MonoKernels run only under Model Runner V2.

---

## comments

**github-actions** (2026-10-09T18:31:16Z):

CC @hongxiayang @tjtanaa @vllmellm @giuseppegrossi for ROCm-related issue

**maeehart** (2026-10-12T16:42:10Z):

[#60731](https://github.com/aarnphm/aarnphm.github.io/issues/60731) ports [#60397](https://github.com/aarnphm/aarnphm.github.io/issues/60397)’s DeepSeek-V4.1 kernels to gfx942 (MI300X and MI325X). Its kernels are in `vllm/models/deepseek_v41/amd/ops/`, and `VLLM_ROCM_MONO_DECODE` from [#60397](https://github.com/aarnphm/aarnphm.github.io/issues/60397) is its only switch, so it can move to `--monokernel` together with [#60397](https://github.com/aarnphm/aarnphm.github.io/issues/60397)’s code. Four points of the port do not fit the RFC as written:

1. **Limits per GPU, not per vendor.** gfx950 takes up to 48 rows at TP2 or TP4. gfx942 takes up to 42 rows at TP4 only, because its workgroup has 64 KB of LDS instead of 160 KB. So `widths` and the refusals need to depend on the device.
2. **A second weight copy.** On gfx942 the kernels cannot read vLLM’s loaded layout, and vLLM’s layers still need their own tensors for prefill and wider steps. So [#60731](https://github.com/aarnphm/aarnphm.github.io/issues/60731) keeps both, about 70 GiB more a rank at TP4. Can the standard allow a declared second copy that is made before memory profiling?
3. **The accuracy reference.** On gfx942 vLLM keeps BF16 activations, while the kernels use MXFP8. Pooled GSM8K shows no loss. If the reference is vLLM’s path on the same GPU, MXFP8 has to become an opt-in on gfx942.
4. **A split layer.** On gfx942 the 8 index layers run a launch, then vLLM’s compressor and indexer, then a second launch. The 3 DSpark draft layers also take a launch, but they are not decoder layers of the target model. The spec needs a mode for the first and a way to name the second.

Also, setting every capture size up to `max(widths)` would help. With 5 DSpark drafts vLLM captures no 30-row graph, so a step of 5 requests runs padded to 36 rows.

**benchislett** (2026-10-13T17:02:35Z):

I don’t particularly like the Monokernel interface designed here. We should probably tidy this up

**Fangzhou-Ai** (2026-10-13T18:52:56Z):

[#61482](https://github.com/aarnphm/aarnphm.github.io/issues/61482) (draft) docks DeepSeek-V4-Pro (TP8, MI355X) onto the merged DeepSeek-V4.1-Flash MonoKernel ([#60397](https://github.com/aarnphm/aarnphm.github.io/issues/60397)). It does this by sharing the kernel code itself, which this RFC and [#61311](https://github.com/aarnphm/aarnphm.github.io/issues/61311) leave to each model:

- [#60397](https://github.com/aarnphm/aarnphm.github.io/issues/60397)’s FlyDSL stages (the mHC seams, the sparse-MLA front and back, the MoE, the in-kernel all-reduce) move to `vllm/models/common/amd/mono/`. A `MonoSpec` (widths, numerics, variant flags read with `const_expr`) parameterizes them.
- Pro adds a spec and a docking class of \~300 lines, with no Pro-only kernel code. V4.1 still builds the same kernels: 48 of 48 variants are byte-identical except for one race fix.
- AgentX on TP8, mono off → on: interactivity P50 +24% at c1, +13% at c4 and +5% at c8. GSM8K is at parity.

How it relates to the RFC:

- The package sits below the runtime layer that [#61311](https://github.com/aarnphm/aarnphm.github.io/issues/61311) proposes (`vllm/models/common/mono/`: routing, fail-stop, guards). It does not replace it.
- The `MonoDecode` class in the PR (deployment checks, step eligibility, runner) maps onto [#61311](https://github.com/aarnphm/aarnphm.github.io/issues/61311)’s per-model glue once that lands.
- The two PRs share the `MonoSpec` name for different things: kernel geometry here, the runtime contract in [#61311](https://github.com/aarnphm/aarnphm.github.io/issues/61311). I’m happy to rename mine (e.g. `KernelSpec`).
- @maeehart: [#60731](https://github.com/aarnphm/aarnphm.github.io/issues/60731) edits the V4.1 files this PR moves. The move is a pure `git mv` plus spec parameterization, so whichever lands second rebases.

**jiacao-amd** (2026-10-14T00:23:43Z):

Related experimental work: [jiacao-amd/vllm#1](https://github.com/jiacao-amd/vllm/pull/1) is a fork-hosted stacked draft against the evaluated [#60683](https://github.com/aarnphm/aarnphm.github.io/issues/60683) revision `9c6c96d8ef`.

The scope is dependency-aware K1/K2 scheduling, sparse wide-K2 routing and producer/consumer panel handoffs for selected speculative-verification shapes through M48. This differs from [#61435](https://github.com/aarnphm/aarnphm.github.io/issues/61435)’s row-tiling and MLA M1 extension, [#60793](https://github.com/aarnphm/aarnphm.github.io/issues/60793)’s standalone MoE fusion, and [#61498](https://github.com/aarnphm/aarnphm.github.io/issues/61498)’s ordinary non-speculative decode support. There is overlapping shape coverage with [#61435](https://github.com/aarnphm/aarnphm.github.io/issues/61435); no direct performance comparison between the approaches has been made.

The draft preserves the measured historical snapshot for review. Porting to the current shared MonoKernel runtime, pruning experimental variants and completing model evaluation remain prerequisites for upstream integration. AI assistance (OpenAI Codex) was used; validation results and limitations are in the draft.

**tlrmchlsmth** (2026-10-15T15:45:25Z):

We should consider requiring some degree compatibility between the monokernels and other implementations.

Cases I’m thinking of:

- If the monokernel is only optimal for decode and not for prefill, we’d want to ensure compatibility for KV transfers.
- Similarly it would be nice to be able to dynamically switch between mono-kernel and non-monokernel implementations based on the batch size.

These would impose constraints on the weight and KV cache layouts

**tlrmchlsmth** (2026-10-15T15:50:33Z):

I think we should most likely require jitted kernels in the monokernels rather than shipping binaries in the wheel in order to keep the wheel size down, since we won’t be able to share binaries between model definitions when using monokernels.

**simondanielsson** (2026-10-16T04:23:45Z):

Happy to get some feedback on the direction of this effort, heavily inspired by @benchislett’s work

[\[Core\] Shared abstractions for specialized decode kernels](https://github.com/vllm-project/vllm/pull/61311)

**waynehacking8** (2026-10-16T13:53:31Z):

I’d like to help with the CUDA side on SM12x (RTX PRO 6000, RTX 5090), which [#61867](https://github.com/aarnphm/aarnphm.github.io/issues/61867)’s Kimi K3 kernels don’t cover since they need SM100/103. On an RTX PRO 6000, nvidia/Qwen3-30B-A3B-NVFP4 decodes at 4.6 ms per token at batch 1 under CUDA graphs, against about 1.6 ms of weight and KV traffic at 80% of peak bandwidth; each step runs about 940 kernels, and about 30% of the step is activation quantization, MoE routing and idle gaps. Would an SM12x consumer of the kit in [#61311](https://github.com/aarnphm/aarnphm.github.io/issues/61311) (a JIT-compiled kernel on vLLM’s paged KV cache, CUDA graph widths 1 to 8, with its `DecodeConformance` tests) be a useful first step? The kit refuses models that carry `@support_torch_compile`, which `Qwen3MoeModel` does, so I’d also like to know whether a separate model definition under `vllm/models/` is the expected route for a model like this.

**Fangzhou-Ai** (2026-10-23T19:03:33Z):

Opened draft [#63398](https://github.com/aarnphm/aarnphm.github.io/issues/63398) for the MiniMax-M3 TP4 MI355X implementation under the existing MRV2 `specialized_decode` lifecycle.

This overlaps with the ATOM fusion work in [#59705](https://github.com/aarnphm/aarnphm.github.io/issues/59705). The submitter discussed its regression versus native with its author and requested this replacement. The draft includes cache rebinding, independent main/index mappings, padded verification and graph replay coverage, bounded cross-rank failure handling, and full-hour AgentX c1/c4 measurements. It does not claim an independent benchmark of [#59705](https://github.com/aarnphm/aarnphm.github.io/issues/59705).

Unlike the approach in [#59653](https://github.com/aarnphm/aarnphm.github.io/issues/59653), the fused sparse-layer kernels and runtime are owned in vLLM, without requiring the proposed external AITER MiniMax fused-layer operator. Existing AITER/FlyDSL primitives are still used. The PR documents the precision change, legacy FFN option, and the qualified `max_num_seqs=2` recipe.

The PR contains test commands and model evaluation results, ATOM attribution, and AI-assistance disclosure. It remains a draft for human code and test review.

**AndreasKaratzas** (2026-10-26T01:45:42Z):

The CI side of this is in draft [#63486](https://github.com/aarnphm/aarnphm.github.io/issues/63486), which I’m reworking into the following shape. It affects how monokernels are structured, so I’m describing it here.

**CI shape**

- One test area, `.buildkite/test_areas/monokernels.yaml`. Every AMD monokernel gets exactly two groups:
  - **Kernel I/O group (blocking, non-optional).** Each fused stage runs on synthetic inputs. Its outputs are compared with a slow, mathematically simple torch reference: plain matmuls, softmax, top-k and RMSNorm, not vLLM’s stock path. The comparison uses a bounded relative error, and refusal cases are included.
  - **End-to-end group (nightly).** It runs gsm8k on the real checkpoint. A check after the eval fails the step if the monokernel’s launch counters did not move on every rank.
- Each group lists curated `source_file_dependencies`: the monokernel, its callers, the model code, the quantization/MoE/loader layouts it reads raw, the AITER wrappers and platform gating it uses, the ROCm image pins, and its tests.
- An `AGENTS.md` rule: a PR that adds an AMD monokernel, whether behind `--monokernel` or `decode_backend`, adds these two groups. If the model already has groups, the PR extends them. The rule also asks for accuracy-oriented asserts with derived tolerances (below), and for both groups to pass at least 10 times in a row before they are submitted.

**Where this agrees with the RFC**

- _Goals 2_ and _Enabling and refusal_: the I/O group asserts that each refusal stage reports every reason. That makes “never a silent fallback” a tested property. It is not one today. With `--monokernel` on, `deepseek_v41/amd/mono_decode.py:403-428` still turns layers off with an `info_once` (for example “needs aiter’s fused mHC seam” or an MoE routing mismatch), and the engine starts anyway.
- _Goal 7_ and [#61311](https://github.com/aarnphm/aarnphm.github.io/issues/61311)’s `test_build_does_not_move_weights`: we’d keep that check and run it on GPU as well.
- _Layout_: I’ll move the DSv4.1 tests under `tests/models/deepseek_v41/amd_specialized_decode/`. Kimi-K3 and MiniMax-M3 already follow that layout.

**Where I’d push back or extend**

1. **Test accuracy, not structure.** Asserts about what was called or built (this class is not that class, this layer object exists, this runner was picked) freeze the current implementation. They break on harmless refactors and pass when the numbers are wrong. Conformance tests should check what the kernel computes, against a reference, with a tolerance that is numerically justified: derived from the input and output formats, the accumulation precision and the reduction length, with the derivation written next to the bound. A threshold widened until CI goes green hides exactly the bugs these tests exist for. Before submitting a group, the author should have run it at least 10 times in a row, so flakiness shows up in the PR rather than in everyone else’s CI.

2. **Testing needs more than CPU tests (_Layout_: “CPU tests for refusals and plan() routing”).** Those tests cover the glue, not the kernels. On a \~9k-line FlyDSL package (`deepseek_v41/amd/mono/`), the main kernel bug our tests found is in a hand-off between stages. DSv4.1’s `stage_combine` reads `klen` without awaiting `krdy`, unlike `stage_split` (`deepseek_v41/amd/mono/attention/back.py:509-521` vs `:260-264`). DSv4-Pro’s copy of the same stage has the await (`deepseek_v4/amd/mono/attention/back.py:602-606`), so the fix landed in one copy only. A shared per-stage I/O test would have caught it in both copies. This also supports @Fangzhou-Ai’s proposal to share the stage code ([#61482](https://github.com/aarnphm/aarnphm.github.io/issues/61482) merged with its own copy under `deepseek_v4/amd/mono/`).

3. **Stage boundaries should be part of the contract.** A spec should expose each fused stage, or each launch in layer mode, as a callable with documented input and output tensors. Tests can then drive a stage in isolation against its reference. Testing only `run()` end to end turns any mismatch into one number for the whole layer, and it hides errors that fall within tolerance. It also gives @tlrmchlsmth’s layout-compatibility point a concrete check: the KV and weight layouts a stage reads become test inputs.

4. **An engagement counter in the contract.** _Goal 4_ lets `plan()` pick the stock path on any step, so a green gsm8k run proves nothing about the monokernel. Today our post-eval check has to read three different internals:

   - DSv4.1: `mono_decode._runner.epoch`
   - Kimi-K3: `decode.runner.epoch`
   - MiniMax-M3: `executor.runner.epoch`, plus a patch on `executor._clear`, because the epoch resets about every 2,000 steps

   I’d like `SpecializedDecode` ([#61311](https://github.com/aarnphm/aarnphm.github.io/issues/61311)) to count kernel launches and skips, with the skips broken down by reason, on every rank.

5. **Bounded waits, not unbounded spins (_Not in Scope_: “If a megakernel fails the engine should stop”).** The engine can only stop if a lost hand-off ends the wait. DSv4.1’s and DSv4-Pro’s `_spin` (`mono/common/sync.py:29-35` in both) is `while pending(v): s_sleep(1)` with no bound, so a hand-off bug hangs the job until the CI timeout. Kimi-K3 (`WAIT_TICKS` plus a health word) and MiniMax-M3 (a 10 s bound plus an error word on every rank) already bound their waits. I’d make that a requirement.

6. **Limits per GPU (@maeehart’s point 1) also apply to CI.** The I/O group has to cover each architecture the spec’s `widths` serves, including the widest width on each. A gfx942 spec without an MI300 step is untested.

7. **JIT (@tlrmchlsmth).** If kernels are JIT-compiled, a build failure only shows up at first launch. The I/O group should therefore compile every variant the spec can select. [#63486](https://github.com/aarnphm/aarnphm.github.io/issues/63486) already does this for Kimi-K3 (all 30 K3 builds).

cc @dllehr-amd

**AndreasKaratzas** (2026-11-08T08:15:51Z):

An update on the CI side ([#63486](https://github.com/aarnphm/aarnphm.github.io/issues/63486)), with measurements. Disk is the main cost in the CI shape I described above, so I’ve added a tier that needs no checkpoints.

**Where the time goes now.** A nightly DeepSeek-V4.1-Flash gsm8k step on MI355X spent about 27 minutes loading the 475 GiB checkpoint and 104 seconds evaluating. That is about 94% of the step’s wall time on disk I/O. Across the four AMD monokernel models, the checkpoints add up to about 3 TB of HF cache. gsm8k is also a weak detector: the configs’ tolerances of 0.024-0.035 are 3 binomial standard errors, so a 2-point regression can pass.

**New tier: teacher-forced logprob parity on random weights (nightly, zero disk).** Each model runs at full width and reduced depth, with `--monokernel` on and off as two engines on the same random weights. Both decode the same prompts through the same forced tokens (`trace_decode_token_ids`), so the arms’ contexts are identical at every position and a difference can’t cascade. The test compares per-position KL, |Δlogprob| of the forced token, and top-1 flips at clear margins. Bounds are about 3x the noise measured over repeated off-vs-off runs. Each test also has to show it can fail: a bug injected into mono’s own weight view must exceed the bounds by at least 10x.

| Model           | Device               | Runtime   | In-test mutant                           |
| --------------- | -------------------- | --------- | ---------------------------------------- |
| DeepSeek-V4-Pro | TP8                  | \~3.6 min | experts’ w2 rolled by one, 74x the bound |
| DeepSeek-V4.1   | TP4                  | \~3.3 min | router off by one, \~120x                |
| MiniMax-M3      | TP4                  | \~2.3 min | two experts’ w2 swapped, 20x             |
| Kimi-K3         | TP8/DCP8, DSpark k=3 | \~1.9 min | MLP AttnRes query negated, 41-43x        |

Things we learned that matter for the standard:

- **vLLM’s dummy weights make logprob comparisons meaningless.** All tensors come from a single seed, so logits are near uniform. The first test drafts showed it: Kimi-K3’s off-vs-off KL (6.7e-3) was the same as its on-vs-off KL (5.7e-3). Each test now uses a realistic random init and asserts the off arm’s entropy and top-1 margin. One trap: seeding by parameter name gives every TP rank the same `lm_head` shard, so every logit appears TP times and every top-1 margin is exactly 0.
- **Teacher forcing alone isn’t enough to remove scheduling noise.** On MiniMax-M3, off-vs-off KL reached 0.043 until the harness held the scheduler until every request was added. With the hold, its p99 is 1.2e-5, the same as with the engine in process.
- **Routing moves are expected, not ties.** On DSv4.1’s random stack, 13-26 of 256 positions change their routed experts between any two runs. These aren’t ties within the router’s rounding; the top-k margins are just small. The test recomputes routing in float64 from each arm’s MoE input, excludes exactly the moved positions, and caps how many may move.
- **It has blind spots.** On random weights, attention contributes about 1% of the residual. RoPE off-by-one, `wo_b` scale errors and single-block E8M0 corruption therefore sit at or below the bounds, and Kimi-K3’s routed experts must be damped to keep the stack from going chaotic, which hides them. So this tier sits between the per-stage float64 I/O tests (blocking) and the real-checkpoint E2E. It does not replace the I/O tests.

**Proposed tiers**

| Tier                                     | Trigger                                                                      | Disk               | Catches                                                                 |
| ---------------------------------------- | ---------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------- |
| Stage I/O vs float64 reference           | per PR, blocking                                                             | 0                  | rounding, scale codes, races between stages, layouts                    |
| Random-weight teacher-forced parity      | nightly                                                                      | 0                  | integration: wiring, routing, residuals, graphs, silent fallback        |
| Real-checkpoint E2E (gsm8k + engagement) | weekly, or on checkpoint / AITER pin bumps, on nodes with a pre-staged cache | \~3 TB, only there | full depth, real activation and scale distributions, drafter acceptance |

**Asks for the contract (in addition to the engagement counter I asked for earlier)**

1. **A re-bind hook in the model-state lifecycle.** The _Layout_ section has `model_state.py` “bind KV caches” once. With the extensible KV cache ([#56492](https://github.com/aarnphm/aarnphm.github.io/issues/56492), on by default), `extend_kv_cache` re-binds every view after warmup at the same addresses with a new block count. Kimi-K3’s AMD decode correctly refuses caches it hasn’t prepared, so its first eager step after startup killed the engine. [#66370](https://github.com/aarnphm/aarnphm.github.io/issues/66370) adds `ModelState.rebind_kv_cache`. The standard should say that a specialized decode prepares on both `initialize_kv_cache` and `rebind_kv_cache`.
2. **The engagement counter is necessary, not optional.** Without it, a parity test whose on arm silently falls back to stock passes as null-vs-null. A reviewer caught exactly that gap in one of our drafts. Each test now asserts an exact launch count per layer and step.
3. **A test-only way to give the mono path its own weight view.** Every mutant here had to be injected through each runtime’s private structures (`_weights`, `MonoLayerWeights`, kernel-launch wrappers). A documented hook, for example `SpecializedDecode.weight_views()`, would let one shared harness inject faults into any model.
4. **A shared realistic random init** (fan-in-scaled normals, per-rank vocabulary shards, varied E8M0 codes) that conformance tests can reuse, instead of each model writing its own.

One question for CI infra: is the AMD HF cache shared across nodes or local to each node? That decides whether the \~3 TB is paid once or on every node, and so how strongly the real-checkpoint tier should be confined to a few tagged nodes.

cc @dllehr-amd

**fxmarty-amd** (2026-11-09T12:42:20Z):

Hi @AndreasKaratzas, I think [vllm-project/vllm#63486](https://github.com/vllm-project/vllm/pull/63486) is quite important.

While testing [vllm-project/vllm#65421](https://github.com/vllm-project/vllm/pull/65421) (see [vllm-project/vllm#65421 (comment)](https://github.com/vllm-project/vllm/pull/65421#issuecomment-6739908867)), it turns out to me that vLLM baseline itself not being deterministic and already heavily quantized in places makes the existing CI/CD tests in various monokernel PRs against vLLM baseline kind of meaningless.

The question PRs and CI/CD should answer is IMO: “is the monokernel path at least as close to reference deterministic high-precision activation-exact baseline than vLLM baseline is?”

and most monkernels PR tests seem to not be confidently answering this question at the moment.

The other questions are:

- should different monokernel implementations (e.g. for different M, different TP config, different precision, arch, etc.) share the same testing infrastructure ?
- should PRs implement FAST single GPU tests to test the correctness ? Or just tests on real TP/EP config is enough ?
- Should there be test groups with `num_devices: 1` & `parallelism: x (e.g. 8 or something high)` with high parallelism, allowing to test single-gpu a little quicker and more comprehensively?
- Similarly, should the `num_devices: 4` / `num_devices: 8` test groups only test e.g. `-m 'distributed(num_gpus=4)'` ?

