---
title: '51855'
pageLayout: default
slug: thoughts/inference/vllm-prs/51855
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/51855.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#51855](https://github.com/aarnphm/aarnphm.github.io/issues/51855) \[K3\] support recoverssm for K3

[vllm-project/vllm#51855](https://github.com/vllm-project/vllm/pull/51855) · ZJY0516 · MERGED · created 2026-08-11T17:49:54Z · merged 2026-08-17T11:34:14Z

<!-- markdownlint-disable -->

## Purpose

This PR adds RecoverSSM, a Kimi-K3-specific state recovery path for speculative KDA decoding on NVIDIA Model Runner V2. It is enabled through `--use-replayssm`, is intended for Kimi-K3 serving with DSpark, and supports `mamba_cache_mode=align` prefix caching.

Without RecoverSSM, KDA speculative decoding materializes a full recurrent state for every speculative position. RecoverSSM keeps one checkpoint and compact per-token records, then reconstructs the state at the accepted position after every verify. Under the tested TP8 Kimi-K3 configuration, this increases the effective cache capacity by 10.97% at the same 46.48 GiB cache-memory budget.

## Design

### Kimi-K3-specific MRV2 integration

The implementation is contained in the Kimi-K3 KDA metadata, layer, and RecoverSSM operator paths. The metadata builder gives speculative rows one KDA checkpoint slot and creates a model-specific post-sampling committer.

RecoverSSM is selected through the existing `--use-replayssm` option. The regular KDA prefill and non-speculative decode paths are unchanged.

### Verify records FP32 corrections

KDA verify reads the checkpoint without modifying it and computes the normal speculative outputs. For each speculative token it records:

- the correction vector `u` in FP32;
- the key and raw gate in the activation dtype.

Recording `u` removes `v` and `beta` from the commit path and avoids recomputing the state-dependent correction. FP32 is used for `u` to preserve reconstruction accuracy; keys and gates remain BF16 in the evaluated configuration.

### Direct accepted-position reconstruction

After verification determines the accepted length, one Triton kernel reconstructs the required checkpoint directly. For the KDA recurrence:



```text
S_t = S_{t-1} * exp(g_t) + u_t outer k_t
```

The accepted state is:



```text
S_p = S_0 * exp(sum(g_1 ... g_p))
    + sum_j u_j outer (k_j * exp(sum(g_{j+1} ... g_p)))
```

The commit kernel evaluates this expression in reverse token order. It loads the original state once and does not recursively update or write intermediate states. The loop bound is the runtime accepted length rather than the complete eight-token DSpark activation window.

In align mode, the same launch also reconstructs the next prefix-cache boundary when the accepted range crosses it. Convolution history is compacted to the accepted position and to the boundary using the same commit plan.

### Cache layout

Each KDA page contains:



```text
(convolution state, KDA checkpoint, FP32 correction records, key/gate records)
```

Recovery records are transient and are not included in prefix-cache state copies. The checkpoint and convolution state remain the prefix-cache boundary state.

## Scope

- NVIDIA Kimi-K3 KDA
- Model Runner V2
- speculative decoding, validated with Kimi-K3-DSpark and seven draft tokens
- `mamba_cache_mode=align` prefix caching

Shared changes are limited to the derived KDA RecoverSSM flag, state-layout helpers, and the post-sampling commit lifecycle. The recovery algorithm and cache layout remain Kimi-K3-specific, and this PR does not add Mamba2 support.

## Test Plan and Results

### Unit tests



```bash
.venv/bin/python -m pytest tests/models/kimi_k3/test_kda.py -v
```

Result: `69 passed`. The 18 RecoverSSM cases cover:

- the actual eight-token DSpark window;
- partial and full acceptance lengths;
- direct final-state and align-boundary reconstruction;
- both convolution-state layouts;
- request-index remapping;
- gates with and without a lower bound;
- cache shape and dtype derivation.



```bash
.venv/bin/pre-commit run --files vllm/model_executor/layers/mamba/mamba_utils.py vllm/models/kimi_k3/nvidia/kda.py vllm/models/kimi_k3/nvidia/ops/recoverssm.py tests/models/kimi_k3/test_kda.py
```

Result: all hooks passed.



```bash
.venv/bin/pre-commit run mypy-3.12 --all-files --hook-stage manual
```

Result: passed for Python 3.12.

### End-to-end accuracy

Controlled setup:

- Kimi-K3 target and Kimi-K3-DSpark draft model, TP8 on two GB300 nodes;
- seven greedy draft tokens and block rejection sampling;
- align-mode Mamba cache, prefix caching, FP8 MLA cache, FlashInfer MLA, and fastsafetensors;
- identical datasets, prompts, scorers, and serving parameters for RecoverSSM ON and OFF.

| GSM8K, 1,319 samples | RecoverSSM ON |      OFF |   ON - OFF |
| -------------------- | ------------: | -------: | ---------: |
| Flexible extract     |      96.3609% | 96.1334% | +0.2274 pp |
| Strict match         |      96.0576% | 95.9060% | +0.1516 pp |

The paired 95% confidence intervals were `[-0.4549, +0.9098]` pp for flexible extract and `[-0.6065, +0.9098]` pp for strict match. Both runs completed all 1,319 requests with zero request errors. Independently initialized greedy serving runs are not bitwise deterministic, so the paired score comparison is used instead of exact generated-text equality.

| MRCR, 40 long prompts run cold and warm | RecoverSSM ON |     OFF |
| --------------------------------------- | ------------: | ------: |
| Cold score                              |       0.74879 | 0.66791 |
| Warm score                              |       0.74879 | 0.66791 |
| Cold/warm exact-output consistency      |          100% |   97.5% |
| Warm prefix-cache token hit rate        |        89.13% |  90.50% |

The MRCR prompts ranged from 16,757 to 33,772 tokens and contained 991,216 prompt tokens per pass. RecoverSSM completed both passes without request errors.

Total DSpark acceptance was 53.3514% with RecoverSSM and 53.4062% without it, a difference of -0.0548 percentage points. GSM8K-only acceptance was 50.0899% with RecoverSSM and 50.0336% without it.

### Memory capacity

Both variants used a fixed 46.48 GiB cache-memory budget.

| Metric                       |    RecoverSSM ON |              OFF | Difference |
| ---------------------------- | ---------------: | ---------------: | ---------: |
| Effective cache capacity     | 2,856,170 tokens | 2,573,777 tokens |    +10.97% |
| Max 262k-request concurrency |          10.895x |           9.818x |    +10.97% |

### Nsight Systems profile

The profile used batch 32, 128 forced output tokens, and 80 captured decode steps per GPU on the same two-node TP8 setup. The aggregate results below include eight initial small-batch steps.

| KDA work per decode step | Initial recurrent recovery | RecoverSSM |
| ------------------------ | -------------------------: | ---------: |
| Verify                   |                   3.115 ms |   3.540 ms |
| Commit                   |                   4.793 ms |   1.464 ms |
| Commit auxiliaries       |                   0.081 ms |   0.081 ms |
| Total                    |                   7.989 ms |   5.085 ms |

The direct reconstruction makes the commit kernel 3.27x faster and reduces total RecoverSSM KDA time by 36.34%. The FP32 correction write increases verify time by 13.66%, but commit time falls by 69.45%. Native KDA without recovery took 2.772 ms per step in the corresponding OFF trace.

The 72 steady batch-32 steps each verified 256 tokens. Their CUDA-graph model-forward GPU wall time was 65.208 ms with RecoverSSM and 64.873 ms without it, a 0.335 ms or 0.52% difference. Kernel residence exceeds wall time because kernels on concurrent streams overlap.

| RecoverSSM ON steady forward kernel residence | Time per step | Share |
| --------------------------------------------- | ------------: | ----: |
| DeepGEMM and nvJet GEMM                       |     38.760 ms | 56.5% |
| MoE routing and packing                       |      8.121 ms | 11.8% |
| SiTU activation                               |      7.682 ms | 11.2% |
| TP collectives                                |      6.136 ms |  8.9% |
| RecoverSSM verify                             |      3.791 ms |  5.5% |
| KDA convolution and normalization             |      0.938 ms |  1.4% |
| MLA attention kernels                         |      0.519 ms |  0.8% |
| Other kernels                                 |      2.658 ms |  3.9% |
| Total kernel residence                        |     68.603 ms |  100% |

RecoverSSM verify runs inside model forward and replaces the native KDA recurrent kernel. Its steady kernel residence was 3.791 ms versus 3.009 ms for native KDA, an increase of 0.782 ms. The smaller 0.335 ms forward wall-time difference reflects kernel overlap and run-to-run variation.

Commit runs after sampling determines the accepted length, so it is not part of model forward. Across the 72 steady batch-32 steps it took 1.711 ms per step and varied with the accepted length.

| Post-sampling RecoverSSM commit      | Time per step | Share |
| ------------------------------------ | ------------: | ----: |
| FP32 correction state reconstruction |      1.618 ms | 94.5% |
| Convolution-state compaction         |      0.081 ms |  4.7% |
| Commit-plan preparation              |     0.0069 ms |  0.4% |
| Align bookkeeping                    |     0.0061 ms |  0.4% |
| Total                                |      1.711 ms |  100% |

Adding the steady model-forward difference and post-sampling commit gives an estimated RecoverSSM GPU critical-path cost of 2.046 ms per batch-32 decode step, before common sampling and postprocessing work.

## Duplicate-work check

The following searches were run against open vLLM PRs:



```bash
gh pr list --repo vllm-project/vllm --state open --search "ReplaySSM Kimi K3"
gh pr list --repo vllm-project/vllm --state open --search "KDA speculative decoding"
gh pr list --repo vllm-project/vllm --state open --search "ReplaySSM in:title,body"
```

No other open PR implements this Kimi-K3 KDA RecoverSSM integration. Related work is materially different:

- [#49847](https://github.com/aarnphm/aarnphm.github.io/issues/49847) targets Mamba2;
- [#49887](https://github.com/aarnphm/aarnphm.github.io/issues/49887) targets Gated DeltaNet;
- [#50140](https://github.com/aarnphm/aarnphm.github.io/issues/50140) is a WIP FlashInfer ReplaySSM kernel;
- [#42792](https://github.com/aarnphm/aarnphm.github.io/issues/42792) adds generic Mamba align-prefix speculative support rather than Kimi-K3 KDA RecoverSSM.

## AI assistance

AI assistance was used for implementation, tests, profiling analysis, model evaluation analysis, and this PR description. The submitting human must review every changed line, reproduce the relevant tests, and understand and defend the design end-to-end.

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [x] The purpose of the PR.
- [x] The test plan and exact commands.
- [x] Unit, end-to-end accuracy, memory, acceptance, and performance results.
- [x] Duplicate-work check and AI-assistance disclosure.

</details>

**BEFORE SUBMITTING, PLEASE READ <https://docs.vllm.ai/en/latest/contributing>**

---

## comments

**mergify** (2026-08-11T17:50:33Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ZJY0516.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-08-13T17:03:23Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ZJY0516.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**ivanium** (2026-08-13T20:20:31Z):

nits: the pr description still says “replayssm” not “recoverssm”

**ZJY0516** (2026-08-15T16:41:23Z):

> it looks like we’re expecting \~3% overhead on the decode step to free up 10% of extra KV cache capacity

The memory waste maybe worse when using DP.

For dp + ep 16 on GB300, model weight needs 190.52 GB, kv cache has 40.97 GB(gpu-memory-utilization 0.9), the KDA state is about 400MB, if you have 4 seqs per rank and each has 7 spec tokens, which means it will waste 12GB memory if don’t use replayssm or recoverssm

I have pruned tests and addressed the pure decode issue. @benchislett

**ZJY0516** (2026-08-17T03:58:09Z):

/ci run

**github-actions** (2026-08-17T03:58:25Z):

✅ Triggered [Buildkite CI #84148](https://buildkite.com/vllm/ci/builds/84148) for commit `51fc7b4535d0`.

**ZJY0516** (2026-08-17T07:16:18Z):

/ci run

**github-actions** (2026-08-17T07:17:51Z):

✅ Triggered [Buildkite CI #84167](https://buildkite.com/vllm/ci/builds/84167) for commit `9c2cd648b492`.

