---
title: '52506'
pageLayout: default
slug: thoughts/inference/vllm-prs/52506
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/52506.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#52506](https://github.com/aarnphm/aarnphm.github.io/issues/52506) \[Mamba\] Add FlashInfer ReplaySSM backend

[vllm-project/vllm#52506](https://github.com/vllm-project/vllm/pull/52506) · askliar · MERGED · created 2026-08-16T08:08:38Z · merged 2026-09-01T14:26:44Z

## Summary

This PR adds FlashInfer as an optional ReplaySSM backend for Mamba2. It calls FlashInfer’s native `checkpointing_ssu` API, shares replay trackers per KV-cache group, and autotunes a maximum-batch decode through the existing FlashInfer warmup flow before CUDA-graph capture. FlashInfer ReplaySSM supports Model Runner V1 and V2; Triton ReplaySSM remains V1-only.

## Blocked on FlashInfer

> \[!tip\]
>
> This PR is blocked by [flashinfer-ai/flashinfer#4545](https://github.com/flashinfer-ai/flashinfer/pull/4545), which adds native ReplaySSM autotuning. Do not merge this PR until [#4545](https://github.com/aarnphm/aarnphm.github.io/issues/4545) is available in a compatible FlashInfer release and vLLM’s package and Docker pins are updated together.

Older FlashInfer versions fail capability validation rather than silently using a non-native tuning path.

## Validation

Validation used `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` on H100 GPUs with MTP disabled.

### GSM8K

Matched 5-shot GSM8K over all 1,319 questions, temperature 0, and `max_num_seqs=64`:

| Backend              |   Correct | Accuracy | Invalid |
| -------------------- | --------: | -------: | ------: |
| FlashInfer ReplaySSM | 1117/1319 |  84.685% |       0 |
| Triton ReplaySSM     | 1119/1319 |  84.837% |       0 |

The paired exact test gives `p=0.913`, so the observed accuracy difference is not statistically significant.

### Throughput

Matched TP=1 fixed-length decode benchmark: 512 decode steps, 128 warmup steps, five measured repeats, and ReplaySSM buffer length 16.

| Batch | FlashInfer TPS | Triton TPS | FlashInfer delta |
| ----: | -------------: | ---------: | ---------------: |
|     1 |         366.22 |     360.17 |           +1.68% |
|    16 |        2774.93 |    2901.00 |           -4.35% |
|    64 |        6630.05 |    6725.68 |           -1.42% |
|   128 |        9368.32 |    9452.60 |           -0.89% |

FlashInfer is faster at batch size 1; Triton remains faster at the larger tested batch sizes.

Additional coverage: V1 and V2 smokes, two-node TP=2 and PP=2 smokes, focused H100 unit/integration tests

---

## comments

**mergify** (2026-08-17T09:09:22Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-08-17T19:20:17Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-08-19T08:56:58Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-08-19T09:34:34Z):

Hi @askliar, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-08-27T18:11:18Z):

Hi @askliar, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-08-28T02:39:47Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-08-28T22:15:50Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**askliar** (2026-08-31T11:59:00Z):

@benchislett I’ve addressed most of the comments - please, let me know if the latest version looks good to you.

Main test runs I’ve completed are (all with N3.5 Lightning NVFP4):

1. GB200/H100 TPS runs for regular single-GPU.
2. GB200/H100 GSM8k validations across different partitioning schemas.

For 1:

### H100

| Baseline TPS |                                  FI ReplaySSM TPS |                          Triton ReplaySSM TPS |
| -----------: | ------------------------------------------------: | --------------------------------------------: |
|       390.50 |     387.69 (-0.72% vs baseline; +1.95% vs Triton) |     380.27 (-3.31% vs baseline; -1.91% vs FI) |
|     2,922.48 |   2,940.48 (+0.62% vs baseline; -3.92% vs Triton) |   3,060.55 (+6.46% vs baseline; +4.08% vs FI) |
|     6,063.88 |   6,575.50 (+8.44% vs baseline; -0.80% vs Triton) |   6,628.40 (+8.68% vs baseline; +0.80% vs FI) |
|     8,203.61 |  9,242.77 (+12.67% vs baseline; -0.45% vs Triton) |  9,284.60 (+12.36% vs baseline; +0.45% vs FI) |
|    11,513.71 | 13,591.81 (+18.05% vs baseline; -0.99% vs Triton) | 13,727.26 (+19.11% vs baseline; +1.00% vs FI) |
|    13,329.24 | 16,434.66 (+23.30% vs baseline; +0.22% vs Triton) | 16,398.29 (+23.51% vs baseline; -0.22% vs FI) |

### GB200

| Baseline TPS |                                  FI ReplaySSM TPS |                         Triton ReplaySSM TPS |
| -----------: | ------------------------------------------------: | -------------------------------------------: |
|       439.27 |     438.16 (-0.25% vs baseline; +5.65% vs Triton) |    414.70 (-7.34% vs baseline; -5.35% vs FI) |
|     3,687.06 |   3,823.27 (+3.69% vs baseline; -0.19% vs Triton) |  3,830.55 (+0.86% vs baseline; +0.19% vs FI) |
|     8,325.10 |   8,743.25 (+5.02% vs baseline; +2.64% vs Triton) |  8,518.40 (+1.53% vs baseline; -2.57% vs FI) |
|    11,963.44 |  12,892.34 (+7.76% vs baseline; +3.42% vs Triton) | 12,466.08 (+3.85% vs baseline; -3.31% vs FI) |
|    17,657.23 | 19,492.70 (+10.39% vs baseline; +3.02% vs Triton) | 18,922.15 (+6.50% vs baseline; -2.93% vs FI) |
|    21,008.20 | 23,847.05 (+13.51% vs baseline; +3.35% vs Triton) | 23,073.70 (+9.38% vs baseline; -3.24% vs FI) |

For 2:

### Distributed GSM8K

| Hardware | Topology | No ReplaySSM | FI ReplaySSM |     Delta |
| -------- | -------- | -----------: | -----------: | --------: |
| H100     | TP2      |      84.230% |      84.003% | -0.227 pp |
| H100     | DP2      |      84.761% |      84.382% | -0.379 pp |
| H100     | DP2+EP   |      84.685% |      84.079% | -0.607 pp |
| GB200    | TP2      |      85.216% |      84.610% | -0.607 pp |
| GB200    | DP2      |      84.230% |      84.534% | +0.303 pp |
| GB200    | DP2+EP   |      84.382% |      84.079% | -0.303 pp |

Note that these differences are within noise range - I did not see any consistent degradation across runs.

**mergify** (2026-09-01T04:48:12Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**askliar** (2026-09-01T06:21:27Z):

Addressed the comment here: [vllm-project/vllm#52928 (comment)](https://github.com/vllm-project/vllm/pull/52928#issuecomment-5488777083)

**mergify** (2026-09-01T06:28:18Z):

Hi @askliar, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**benchislett** (2026-09-01T10:57:19Z):

/ci run

**github-actions** (2026-09-01T10:57:38Z):

✅ Triggered [Buildkite CI #86567](https://buildkite.com/vllm/ci/builds/86567) for commit `00a1d2ec9f96`.

**github-actions** (2026-09-01T12:44:34Z):

❌ @askliar, A reviewer with write access must run `/ci run`, approve the PR, or add the `ready` label first.

<!-- vllm-ci-command:5494130746 -->

**askliar** (2026-09-01T12:44:57Z):

/ci run

**github-actions** (2026-09-01T12:45:50Z):

❌ @askliar, A reviewer with write access must run `/ci run`, approve the PR, or add the `ready` label first.

<!-- vllm-ci-command:5494138569 -->

**github-actions** (2026-09-01T12:47:08Z):

✅ @askliar, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**askliar** (2026-09-01T12:47:47Z):

/ci retry

**github-actions** (2026-09-01T12:48:04Z):

✅ The previous CI build is still running: <https://buildkite.com/vllm/ci/builds/86567>

**askliar** (2026-09-01T13:00:41Z):

/ci retry

**github-actions** (2026-09-01T13:01:01Z):

✅ Triggered [Buildkite CI #86602](https://buildkite.com/vllm/ci/builds/86602) for commit `dc9fc45e0574`, running 2 failed step(s) from [Buildkite CI #86567](https://buildkite.com/vllm/ci/builds/86567).

**askliar** (2026-09-01T13:03:05Z):

/ci retry

**github-actions** (2026-09-01T13:03:25Z):

✅ Queued 1 failed job(s) for retry in [Buildkite CI #86602](https://buildkite.com/vllm/ci/builds/86602).

**askliar** (2026-09-01T13:05:45Z):

/ci run

**github-actions** (2026-09-01T13:06:04Z):

✅ Triggered [Buildkite CI #86603](https://buildkite.com/vllm/ci/builds/86603) for commit `dc9fc45e0574`.

