---
title: '60515'
pageLayout: default
slug: thoughts/inference/vllm-prs/60515
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/60515.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#60515](https://github.com/aarnphm/aarnphm.github.io/issues/60515) \[CI\]\[Mamba\] Use well-conditioned prompts for FlashInfer ReplaySSM parity tests

[vllm-project/vllm#60515](https://github.com/vllm-project/vllm/pull/60515) · ZJY0516 · MERGED · created 2026-10-08T01:36:50Z · merged 2026-10-08T09:39:56Z

## Overview

Fix the persistently failing `test_replayssm_flashinfer_decode_matches_baseline[v1-nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4]` in the `(H100) ReplaySSM E2E` job. The failure comes from an ill-conditioned prompt, not ReplaySSM: the FlashInfer parity tests now use a prompt that decodes confidently past the replay window. Test-only change.

## Claims

- The FlashInfer ReplaySSM decode (MRV1 + MRV2) and spec-decode parity tests no longer depend on a near-tie in `"Once upon a time, in a small village,"`.
- The Triton/Nano tests keep the original `PROMPTS`; no non-test code changes.

## Validation

Local, 1x H200 (SM90, same as the CI H100), FlashInfer 0.7.0.post1.

Before (reproduces CI):



```
pytest v1/e2e/test_replayssm_decode.py::test_replayssm_flashinfer_decode_matches_baseline -k v1
FAILED ...[v1-...NVFP4] - AssertionError: Test1:
  Matched tokens: [2156, 1486, 1261, 6552, 8401, 8876, 52206, 2274, 20127, 1317, 3346, 1046]
  baseline: ' there was a young girl named Maya who loved to read. a young girl ...'  {' a': -0.68, ...}
  replayssm_flashinfer: ' there was a young girl named Maya who loved to read. apart from ...'  {' apart': -3.51, ...}
```

After:

| Test                                                        | Result |
| ----------------------------------------------------------- | ------ |
| `test_replayssm_flashinfer_decode_matches_baseline[v1-...]` | passed |
| `test_replayssm_flashinfer_decode_matches_baseline[v2-...]` | passed |
| `test_replayssm_flashinfer_spec_decode_matches_baseline`    | passed |

Prompt stability across 5 separately started engines (3 baseline, 2 ReplaySSM; same config as `_check_replayssm_parity`, CUDA graphs on), compared pairwise (10 pairs) with the `check_logprobs_close` rule:

| Prompt                                                         | Pairs failing                              | Min top-1/top-2 margin |
| -------------------------------------------------------------- | ------------------------------------------ | ---------------------- |
| `"Once upon a time, in a small village,"` (old)                | 2/10, **including ReplaySSM vs ReplaySSM** | 0.0 (token 1)          |
| `"The capital of France is"`                                   | 0/10                                       | 0.0–0.5                |
| `"It was the best of times, it was the worst of times,"` (new) | 0/10, exact match in all 10                | ≥ 1.25                 |

## Details

**Root cause.** The NVFP4 Nemotron-3.5 checkpoint does not produce bitwise-identical logits across engine instances, even with an identical baseline config: three baseline-only engines disagreed on prompt 1 from the first generated token (`" there was"` vs `" there lived"`, margin 0–0.12). The `" there was"` branch falls into a degenerate repetition (`"...loved to read. a young girl named Maya..."`) with a nearly flat next-token distribution, where run-to-run noise pushes one side’s token out of the other side’s top-5. Two identically configured ReplaySSM engines fail `check_logprobs_close` against each other the same way, and baseline-vs-ReplaySSM divergence is within baseline-vs-baseline divergence, so ReplaySSM numerics are not the cause. I also checked that vLLM’s ring-tracker advance matches the FlashInfer `checkpointing_ssu` flush rule (`prev_k + T > MAX_WINDOW`).

**Fix.** Add `FLASHINFER_PROMPTS`, replacing the fragile prompt with `"It was the best of times, it was the worst of times,"`, which decodes all 32 tokens confidently and so still exercises the 16-token window flush/replay. The spec-decode parity test uses the same model and prompts and is switched too. `test_replayssm_flashinfer_mtp` only checks output length and is unchanged.

**Limitations / follow-up.** The underlying run-to-run non-determinism of the NVFP4 baseline (likely MoE kernel/autotune selection) is not addressed here.

**Duplicate check.** Searched issues/PRs for `test_replayssm_decode`, `replayssm_flashinfer_decode_matches_baseline`, and related keywords: none found. Of the open PRs touching this test file ([#46239](https://github.com/aarnphm/aarnphm.github.io/issues/46239), [#47576](https://github.com/aarnphm/aarnphm.github.io/issues/47576), [#48792](https://github.com/aarnphm/aarnphm.github.io/issues/48792), [#49847](https://github.com/aarnphm/aarnphm.github.io/issues/49847), [#49887](https://github.com/aarnphm/aarnphm.github.io/issues/49887), [#50380](https://github.com/aarnphm/aarnphm.github.io/issues/50380), [#52962](https://github.com/aarnphm/aarnphm.github.io/issues/52962), [#55688](https://github.com/aarnphm/aarnphm.github.io/issues/55688)), none changes these prompts or addresses this failure. [#55688](https://github.com/aarnphm/aarnphm.github.io/issues/55688) restructures the file’s constants and may need a trivial rebase.

---

<details>
<summary> Pull Request Checklist </summary>

- [x] I used vLLM’s `/pr-checklist` skill. (Mandatory for agents, optional for humans).

- [x] AI assistance was used during the creation of this PR.

- [x] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

- [x] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

- [x] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.

- [x] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**ZJY0516** (2026-10-08T01:40:37Z):

/ci run —allow-stale

**github-actions** (2026-10-08T01:40:57Z):

✅ Triggered [Buildkite CI #93482](https://buildkite.com/vllm/ci/builds/93482) for commit `c4e8aa357ea8`.

⚠️ This PR is 34 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

**vllm-agent** (2026-10-08T02:02:01Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 1 test steps (1 jobs) instead of 2 (2 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 2 (2)         | 1 (1)    | 1 (1)      | 0 (0)     |
| AMD mirrors            | 0 (0)         | 0 (0)    | 0 (0)      | 0 (0)     |

<details><summary>Selector would run (1)</summary>

- `replayssm-e2e`

</details>

<details><summary>Would skip (today's rules run them) (1)</summary>

- `v1-others-cpu`

</details>

<details><summary>Would add (today's rules do not run them) (0)</summary>

none

</details>

<details><summary>AMD mirrors: would skip (0)</summary>

none

</details>

<details><summary>AMD mirrors: would add (0)</summary>

none

</details>

<sub>1 changed files · base `e58f8dde7f` · head `f118eb5d6a` · Python record: build 93039 at `1e5d0ea888` · kernel record: table [`a982c81`](https://github.com/aarnphm/aarnphm.github.io/commit/a982c81e3e) (build 93415), map [`a982c81`](https://github.com/aarnphm/aarnphm.github.io/commit/a982c81e3e) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 0 optional steps the selector would also run</sub>

**ZJY0516** (2026-10-08T07:05:42Z):

/ci run —allow-stale

**github-actions** (2026-10-08T07:06:03Z):

✅ Triggered [Buildkite CI #93538](https://buildkite.com/vllm/ci/builds/93538) for commit `f118eb5d6a60`.

