---
title: '47572'
pageLayout: default
slug: thoughts/inference/vllm-prs/47572
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/47572.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#47572](https://github.com/aarnphm/aarnphm.github.io/issues/47572) \[RFC\]: ReplaySSM: cache SSM inputs instead of state for faster standard and speculative decode (Mamba2 + GDN)

[vllm-project/vllm#47572](https://github.com/vllm-project/vllm/issues/47572) · Johnny-Liou · OPEN · created 2026-07-03T19:56:53Z

### Motivation.

## Summary

[ReplaySSM](https://dao-lab.ai/blog/2026/replayssm/) is a memory-bandwidth optimization for the decode kernels of state space model (SSM) and linear attention layers. It is a collaboration between Dao AI Lab and NVIDIA; a concurrent cross-framework implementation is in the [TensorRT-LLM PR](https://github.com/NVIDIA/TensorRT-LLM/pull/14203).

SSM decode is memory-bound: today each step reads and writes back the full recurrent state. ReplaySSM instead caches recent inputs, reconstructing the state only when needed and otherwise computing the output directly from the cache. This keeps most steps read-only and writes the full state back only once every L steps.

ReplaySSM provides three benefits:

1. Memory-traffic reduction in a bandwidth-bound kernel: the per-step state write is replaced by a small input append.
2. Cheap rollback for speculative decoding: caching recent SSM inputs makes rollback a buffer pointer move, avoiding the separate per-draft-token state the current implementation keeps.
3. Parallel verification for speculative decoding: on non-flush steps the output is read directly without materializing the state, so a draft window is verified in one shot instead of by a serial recurrence.

Implementation: we have opened a draft PR ([#47576](https://github.com/aarnphm/aarnphm.github.io/issues/47576)) implementing ReplaySSM in vLLM, based on the [ReplaySSM reference repository](https://github.com/Johnny-Liou/ReplaySSM).

We are aware of the existing feature request [#46187](https://github.com/aarnphm/aarnphm.github.io/issues/46187), which tracks ReplaySSM in vLLM and is assigned to us as the ReplaySSM authors. This RFC and the accompanying draft PR are the upstream contribution we committed to.

## Motivation

Two problems motivate ReplaySSM.

1. In hybrid models, the SSM (linear-attention) layers are the decode bottleneck at short-to-medium output lengths (Fig 2 in the blog post). SSM decode is memory-bound and its per-step state read/write dominates HBM traffic.
2. Speculative decoding must roll back rejected draft tokens, but the SSM state update is irreversible: once history is summarized into the recurrent state, a token cannot be subtracted back out. To support rollback, the current implementation keeps a separate recurrent state per draft token, which costs extra memory traffic and extra HBM footprint (about (1 + num_spec)× the state), hurting both latency and maximum concurrency.

ReplaySSM addresses both by caching inputs instead of state. It keeps a single checkpoint state S0 and caches the recent SSM inputs in a ring buffer; on a flush step (once every L steps) it folds the cached inputs into the checkpoint, and on every other step it computes the output directly from the checkpoint plus the cached inputs without materializing the full state.

For standard decode, writing the state back only once every L steps (instead of every step) roughly halves the dominant memory traffic. For speculative decode, caching recent inputs makes rollback a ring-buffer pointer move, cutting the per-step rollback cost from O(T) to O(1) (`T = num_spec_tokens + 1`) and removing the per-draft-token state storage, which improves both latency and maximum concurrency.

ReplaySSM speeds up the SSM kernel most at large serving batch sizes, where the kernel is fully memory-bound (small batch is not a relevant operating point: there the kernel is overhead-bound rather than memory-bound, and e2e latency is dominated by the MLP layers). For hybrid models it gives the largest end-to-end speedup at short-to-medium output lengths, where the SSM layers dominate latency.

### Proposed Change.

## Proposed Change

We propose adding ReplaySSM to vLLM as an opt-in decode path for hybrid SSM models, covering both SSM families and both decode modes, with the baseline path unchanged when the feature is off.

### Scope

- Mamba2 and Gated DeltaNet (GDN) SSM layers.
- Standard and speculative decode.
- New Triton kernels plus the vLLM integration (mixer dispatch, attention-metadata cursors, state-shape sizing). The conv path reuses vLLM’s existing `causal_conv1d_update` unchanged.

### Integration and backward compatibility

The feature is opt-in through config flags that all default to off; when off, the decode path is byte-for-byte the baseline. The ReplaySSM caches are added as extra tensors on the existing paged mamba state, so they reuse vLLM’s paged-block allocation and block-table indexing (continuous batching, per-request allocation and eviction).

### Config flags (all default off)

- `--use-replayssm`: Mamba2/GDN standard cached decode.
- `--use-replayssm-spec`: speculative cached decode (requires speculative decoding and `--mamba-cache-mode none`).
- `--replayssm-buffer-len`: history buffer length (default 16).
- `--replayssm-route` (Mamba2 standard only): `output_only` (default) or `state_and_output`; a temporary knob for the precision study, planned for removal (see Open questions).

### Key integration points

- Paged cache: the input caches and the checkpoint state live in the paged KV block, indexed per request by physical block id; only small per-block cursors live in the metadata builder.
- CUDA graphs and continuous batching: speculative decode uses two device-routed launches (verify and flush) with the flush decision and rollback computed on-device, so the launch sequence is fixed and one captured graph per batch size covers the path.
- What we cache: only the state-building inputs; the output read vector (Mamba2 C, GDN q) is taken from the current activations and never cached.
- Flush decision (spec): we flush when `history + T > B`, where `B` is the history buffer length, and size the ring buffer to `B + T`. This is a slight change from the formulation in the blog post.

### Validation

Correctness is checked at three levels (per-kernel vs an exact reference, prefill-vs-decode equivalence over full sequences, and engine-level parity via `check_logprobs_close`), plus a speculative-decode acceptance-length match on GSM8K. Performance covers standard decode (kernel and end-to-end) and speculative decode (throughput and maximum concurrency). Full numbers are in the draft PR.

## Performance

Please check the detailed experiment results in the draft PR ([#47576](https://github.com/aarnphm/aarnphm.github.io/issues/47576)). Here are brief results.

### Standard decode

Whole-model per-step speedup (baseline / ReplaySSM), fp32 state, buffer length 16:

| Model (SSM, HW)                    | bs 1 | bs 32 | bs 128 | bs 512 |
| ---------------------------------- | ---- | ----- | ------ | ------ |
| Nemotron-Nano-4B (Mamba2, H100)    | 0.99 | 1.17  | 1.40   | 1.55   |
| Qwen3.5-4B (GDN, H100)             | 0.99 | 1.08  | 1.21   | 1.31   |
| Nemotron-Super-120B (Mamba2, B300) | 0.98 | 1.10  | 1.27   | 1.67   |
| Qwen3.5-122B (GDN, B300)           | 0.98 | 1.07  | 1.14   | 1.20   |

### Speculative decode

End-to-end throughput at (B, T) = (16, 4), on B300 with an MTP drafter. `c/std` = ReplaySSM-spec / baseline-spec throughput; `c/AR` = ReplaySSM-spec / no-spec autoregressive throughput.

ReplaySSM-spec vs native-spec (`c/std`):

| Model                        | bs 1 | bs 32 | bs 256 |
| ---------------------------- | ---- | ----- | ------ |
| Nemotron-Super-120B (Mamba2) | 1.09 | 1.07  | 1.88   |
| Qwen3.5-122B (GDN)           | 0.99 | 1.19  | 1.80   |

ReplaySSM-spec vs no-spec AR (`c/AR`):

| Model                        | bs 1 | bs 32 | bs 256 |
| ---------------------------- | ---- | ----- | ------ |
| Nemotron-Super-120B (Mamba2) | 2.12 | 1.90  | 2.21   |
| Qwen3.5-122B (GDN)           | 1.90 | 1.93  | 1.85   |

### Feedback Period.

## Feedback Period

One week from posting; we will keep it open longer if discussion is active.

### CC List.

_No response_

### Any Other Things.

## Work in progress

1. Tensor parallelism test files (e.g., TP1==TP2) and evaluation.
2. Evaluation on lower-precision (fp16/fp8) state. Current evaluation focus on FP32 state (default in most hybrid models)
3. Numerical-precision benefit: because reconstruction is FP32 and the state is written back only on flushes, a low-precision checkpoint (e.g. FP16) is re-quantized \~`L`× less often than the per-step baseline. Reference: [NVIDIA Nemotron-3-Ultra Technical Report](https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf), Section 4.5.

## Open questions

1. Timeline: We plan to split the integration into three stages: Mamba2 standard decode <span>&rarr;</span> GDN standard decode <span>&rarr;</span> Mamba2/GDN speculative decode.
2. Paged-cache layout: is appending ReplaySSM tensors onto the mamba state the pattern you prefer, or should the caches be a separate allocation?

### Before submitting a new issue…

- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.

---

## comments

**ZJY0516** (2026-08-01T06:52:32Z):

How does it work with prefix caching + spec decoding?

**Johnny-Liou** (2026-08-01T19:00:01Z):

Hi @ZJY0516, we will not include this in the first speculative decoding PR, but we would like to share the high-level concept. This is actually the same `unable to rollback` challenge for speculative decoding, and we could use the same approach.

For ReplaySSM, each page includes conv_state, ssm_state, and ring_cache (e.g., dt/b/x in Mamba2). Instead of representing a cached state using only conv_state and ssm_state, we can store conv_state, ssm_state, and ring_cache.

For example, suppose we want to cache the state at the 100th token. Instead of performing an additional flush after the verification step, we can store the checkpointed state at the 98th token together with two tokens in the ring cache for prefix caching.

When this checkpointed state is hit, we can prepend the two tokens to the remaining prompt during prefill.

**ZJY0516** (2026-08-03T03:05:48Z):

> For example, suppose we want to cache the state at the 100th token. Instead of performing an additional flush after the verification step, we can store the checkpointed state at the 98th token together with two tokens in the ring cache for prefix caching.

I don’t think it’s a good idea because it would add a lot of complexity to the KV cache manager, and the ring cache isn’t managed by it now

**github-actions** (2026-11-02T01:36:12Z):

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

**Rockyeast** (2026-11-07T00:26:12Z):

Before opening a PR, I’d like to check whether a focused CUDA change is welcome here, since it overlaps with work in progress.

**What I have.** A Hopper kernel for the existing fused GDN spec decode path. It stores the recurrent state once per step and keeps a small record per later token, and the next step replays the accepted records inside the same verify kernel. It runs for FP32 state, with prefix caching on or off, and needs no new flag. With prefix caching, a step close to a block boundary stores per token instead, so the existing boundary copies work unchanged.

**How it relates to open work.**

|                                                                                                                                            | Path                        | How it is enabled                        | Commit step               | Prefix caching                                               |
| ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | ---------------------------------------- | ------------------------- | ------------------------------------------------------------ |
| [#61434](https://github.com/aarnphm/aarnphm.github.io/issues/61434), [#64325](https://github.com/aarnphm/aarnphm.github.io/issues/64325)   | Triton                      | automatic on that path                   | replay in the next verify | not supported                                                |
| [#59366](https://github.com/aarnphm/aarnphm.github.io/issues/59366) on [#58863](https://github.com/aarnphm/aarnphm.github.io/issues/58863) | CUDA fused, plus RecoverSSM | `--use-replayssm`, declared for Qwen4Exp | separate commit kernel    | align mode supported                                         |
| This change                                                                                                                                | CUDA fused, Hopper only     | automatic with FP32 state                | replay in the next verify | align mode supported, per-token stores near block boundaries |

The record and the update are the same idea as [#59366](https://github.com/aarnphm/aarnphm.github.io/issues/59366) (`h = h * decay + k * delta`), and [#59366](https://github.com/aarnphm/aarnphm.github.io/issues/59366) came first. This change does not add RecoverSSM, and it covers less: FP32 state only, SM90 only, at most 7 draft tokens. What differs is that it needs no new flag and no RecoverSSM, and replays without a separate commit launch. With align support it now overlaps [#59366](https://github.com/aarnphm/aarnphm.github.io/issues/59366) more than my first version of this comment said, so the question below matters more.

**Kernel time** (H200, K=V=128, HV=48, H=16, 8 tokens per request, about 5.4 accepted, 48 layers of state, CUDA graph, µs per layer). The [#59366](https://github.com/aarnphm/aarnphm.github.io/issues/59366) commit skips the conv part, and FlashInfer does not include the gated RMSNorm, so both are slightly favoured. Measured before align support was added.

| requests | fused CUDA (main) | FlashInfer 0.7 `gated_delta_rule_mtp` | [#59366](https://github.com/aarnphm/aarnphm.github.io/issues/59366) verify + commit | this kernel |
| -------: | ----------------: | ------------------------------------: | ----------------------------------------------------------------------------------: | ----------: |
|        1 |              24.8 |                                  17.0 |                                                                   19.7 + 2.6 = 22.3 |        12.2 |
|        8 |              73.7 |                                  81.7 |                                                                  40.5 + 18.3 = 58.7 |        40.1 |
|       32 |             277.5 |                                 275.6 |                                                                140.5 + 74.6 = 214.8 |       135.6 |
|       64 |             542.0 |                                 537.7 |                                                               270.1 + 151.7 = 422.1 |       258.4 |

**End to end** (H200 TP1, `Qwen/Qwen3.8-27B-FP8` + `z-lab/Qwen3.8-27B-DFlash2`, 7 draft tokens, ShareGPT, 512 output tokens with `--ignore-eos`, stock then this change on one GPU, one run per point):

| concurrency | default flags: main → this change | `--no-enable-prefix-caching`: main → this change |
| ----------: | --------------------------------- | ------------------------------------------------ |
|           1 | 281.0 → 289.3 (+3.0%)             | 303.7 → 317.4 (+4.5%)                            |
|           8 | 1567.5 → 1668.6 (+6.4%)           | 1648.4 → 1832.0 (+11.1%)                         |
|          32 | 2843.6 → 3328.7 (+17.1%)          | 3025.1 → 3640.3 (+20.3%)                         |
|          64 | 3066.9 → 3624.4 (+18.2%)          | 3290.8 → 3951.8 (+20.1%)                         |

The two columns come from different machines, so compare within a column. The right column was measured before align support was added. An earlier run gave +8.8% at concurrency 1 there, so that point moves by a few percent between runs.

GSM8K with default flags (512 max tokens): 5-shot, 1319 questions, 0.939 → 0.934; 12-shot, 400 questions, 65% prefix cache hits, 0.927 → 0.927. With 701 forced preemptions: 0.935.

**Question for the maintainers and the authors of [#59366](https://github.com/aarnphm/aarnphm.github.io/issues/59366) and [#61434](https://github.com/aarnphm/aarnphm.github.io/issues/61434):** would you prefer this as its own PR for the default Hopper path, or folded into the RecoverSSM work? I’m happy to do either, and I won’t open a PR until it’s clear this isn’t duplicating effort.

AI assistance was used for this work. I reviewed the change and ran the tests myself.

Edited 2026-11-06: the change now supports prefix caching, and the end-to-end numbers are new, on the wheel of the base commit instead of an older nightly.

