---
title: '52988'
pageLayout: default
slug: thoughts/inference/vllm-prs/52988
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/52988.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#52988](https://github.com/aarnphm/aarnphm.github.io/issues/52988) \[Spec decode\] Support variable-length decode for Kimi-K3 adaptive ver

[vllm-project/vllm#52988](https://github.com/vllm-project/vllm/pull/52988) · qiching · MERGED · created 2026-08-19T18:00:52Z · merged 2026-09-23T03:43:33Z

## Summary

Make Kimi-K3’s MLA and KDA decode paths capturable as FULL varlen CUDA graphs so they work under adaptive DSpark verification ([#47808](https://github.com/aarnphm/aarnphm.github.io/issues/47808)), which schedules ragged per-request draft budgets and requires device-sourced query lengths. Before this change K3’s `FLASHINFER_MLA` reported `UNIFORM_BATCH`/`UNIFORM`, and the decode builders derived per-request query length from a measured/averaged count, so a graph captured on a uniform dummy did not match ragged replay.

## Changes

**FlashInfer MLA** (`flashinfer_mla.py`)

- Promote `_cudagraph_support` to `AttentionCGSupport.ALWAYS` and `query_len_support` to `QueryLenSupport.VARLEN`.
- On the causal spec-decode path, pass `cum_seq_lens_q` + `max_q_len` to `trtllm_batch_decode_with_kv_cache_mla` (flashinfer [#3238](https://github.com/aarnphm/aarnphm.github.io/issues/3238)); assert this path runs on trtllm-gen and does not return LSE (mutually exclusive with DCP).
- Flatten non-causal and DCP decode rows by mapping each row to its request with `searchsorted` on the device query offsets instead of `repeat_interleave(uniform_len)`.
- Carry `query_start_loc_device`, `cum_seq_lens_q`, and `max_query_len` on the decode metadata; set `cum_seq_lens_q` only when `max_query_len > 1`.

**MLA common** (`mla_attention.py` + subclass builders)

- Thread a promised `max_query_len` through `MLACommonMetadataBuilder._build_decode` (clamped to `reorder_batch_threshold`) and the `flashmla`, `flashattn_mla`, `rocm_aiter_mla`, and `dots3_note` builders.

**KDA** (`kda_metadata.py`)

- Promote `KimiK3KDAMetadataBuilder` to `AttentionCGSupport.ALWAYS`.
- Route every active decode through the spec path by masking on `is_prefilling` rather than draft count.
- Return `True` from `supports_device_cpu_query_lens_mismatch`.

**KDA RecoverSSM** (`kda_metadata.py`, opt-in via `--use-replayssm`, off by default)

- Make RecoverSSM co-exist with the adaptive path so it can drop the KDA spec-decode SSM state from `num_spec+1` slots to 1 per request (kernels already in [#51855](https://github.com/aarnphm/aarnphm.github.io/issues/51855)):
  - Restore `self.layer_names` on the builder so RecoverSSM’s commit context can resolve the per-layer forward context (the GDN base does not retain it; inert when RecoverSSM is off).
  - Demote spec rows whose `query_len` exceeds `num_spec+1` out of the RecoverSSM path instead of raising. Adaptive’s cost-table profiler probes token counts past the cudagraph limit by even-splitting them over the capped request slots, producing eager dummy rows longer than RecoverSSM’s `num_spec+1` verify buffers; real captured/serving decode rows are always `<= num_spec+1`, so they are unaffected.

**Mamba-hybrid model state** (`mamba_hybrid.py`)

- Prefer the batch’s promised `max_query_len`, and classify spec-decode rows off `is_prefilling` rather than `num_scheduled_tokens == draft_count + 1`.

**K3 DSpark draft** (`dspark_mla.py`)

- Build the confidence head when the config enables it, load its weights (skipped only when the draft did not build one), and expose `compute_confidence`.

Related Issue: [vllm-project/vllm#51867](https://github.com/vllm-project/vllm/issues/51867)

## Test plan

- [x] Updated the MLA backend metadata tests for the new `_build_decode`
  signature (`test_mla_backends.py`, `test_rocm_aiter_mla_mtp_split.py`).
- [x] Adaptive DSpark spec decode on Kimi-K3 (TP8, fp8 KV).
- [x] RecoverSSM + adaptive on Kimi-K3: boots and serves through full-graph capture; validated against the base-adaptive checks (accuracy/coherence artifacts were overwritten by later runs and are being regenerated).

## Validation

Kimi-K3 (target) + `Inferact/Kimi-K3-DSpark` (draft), TP8 on 8×B300, FP8 KV cache, `attention_backend=FLASHINFER_MLA`, `num_speculative_tokens=7`, `enable_adaptive_verification=true`. (Numbers below are RecoverSSM off; it is opt-in.)

### Accuracy — lossless vs. no-spec (GSM8K, in-tree runner, 1319 questions, 5-shot, temperature 0)

| max-concurrency | no-spec | adaptive |
| --------------- | ------- | -------- |
| 16              | 0.949   | 0.949    |
| 64              | 0.951   | 0.949    |
| 128             | 0.948   | 0.948    |

Adaptive matches no-spec within run-to-run noise (invalid-response rate \~0.1% in every
run), i.e. adaptive verification is lossless at temperature 0.

### Coherence (MTBench, 80 prompts, temperature 1)

Adaptive: 80/80 successful requests, 0 failed, \~19.5K generated tokens; no repetition or degeneration in the saved completions.

### Speculative-decoding health (speed_bench, temperature 1, c16)

Acceptance rate 28.95%, acceptance length 3.03, per-position acceptance decaying 74.5% (pos 0) → 6.9% (pos 6) — non-zero at every position, confirming the draft + confidence head + varlen verification path is exercised end to end.

### RecoverSSM (opt-in) health

Adaptive + RecoverSSM serves through full-graph capture with healthy acceptance length (2.4–2.9 across c1–c256), confirming the RecoverSSM verify/commit path is exercised end to end; matches base-adaptive accuracy/coherence.

## Benchmarking

Pareto sweep of the varlen decode path: 16 configurations × 9 concurrency levels = 144 serving runs. Configurations are fixed draft length K=1..7 plus adaptive, each with and without RecoverSSM.

Kimi-K3 (target) + `Inferact/Kimi-K3-DSpark` (draft), TP8 on 8×B300, FP8 KV cache, `attention_backend=FLASHINFER_MLA`, `draft_sample_method=probabilistic`, `num_speculative_tokens=7` for adaptive. `speed_bench` qualitative, `--speed-bench-output-len 2048`, `--max-model-len 16384`, prompt count scaled with concurrency. Axes: per-user token rate (`1000 / median TPOT`) vs aggregate output throughput.
<img width="1870" height="1275" alt="pareto_throughput_vs_tps_per_user" src="https://github.com/user-attachments/assets/6d8b1237-65ee-4ecd-9328-02eaa3c0e457" />

Overlaying both families lets a non-RecoverSSM point appear to overtake a RecoverSSM one it never competes with, so each family is also plotted with its own frontier.

<img width="1870" height="1275" alt="pareto_no_recoverssm" src="https://github.com/user-attachments/assets/715e0e75-479d-42d9-8f60-470d1f351db6" />
<img width="1870" height="1275" alt="pareto_recoverssm" src="https://github.com/user-attachments/assets/583a02b0-644c-4d77-b455-b12ae6e6c9d7" />

### RecoverSSM, adaptive off vs on

| c   | off    | on     | gain    |
| --- | ------ | ------ | ------- |
| 64  | 2248.6 | 2349.4 | +4.5%   |
| 128 | 2241.5 | 4454.7 | +98.7%  |
| 256 | 2222.9 | 4887.9 | +119.9% |

---

## comments

**mergify** (2026-08-20T20:12:21Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @qiching.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**TheEpicDolphin** (2026-08-29T02:01:51Z):

Hi @qiching, would you mind rebasing?

**github-actions** (2026-09-16T01:10:12Z):

✅ @qiching, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**TheEpicDolphin** (2026-09-16T01:10:14Z):

/ci run

**github-actions** (2026-09-16T01:10:30Z):

❌ This PR is 1335 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5690498731 -->

**qiching** (2026-09-16T23:21:56Z):

> Thanks for the updates @qiching. The `_cudagraph_support = ALWAYS` change will likely cause degeneration for —cudagraph-mode FULL when adaptive verification is disabled. However, I realized that this issue is present for some of DSV4’s backends as well (was able to trigger it with DeepSeek-V4-Flash). Given that it’s a pre-existing issue, I think I’ll just follow up with a fix in a separate PR to address all affected backends.
>
> I don’t want to block this PR on the above issue. LGTM

thanks!

**TheEpicDolphin** (2026-09-16T23:49:50Z):

@qiching when you get the chance, can you please click rebase and then comment “/ci run” to trigger CI?

**qiching** (2026-09-17T19:17:33Z):

> @qiching when you get the chance, can you please click rebase and then comment “/ci run” to trigger CI?

yes. let me do it.

**benchislett** (2026-09-21T15:46:05Z):

/ci run

**github-actions** (2026-09-21T15:46:25Z):

❌ This PR is 53 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5763289080 -->

**benchislett** (2026-09-21T15:47:03Z):

/ci run

**github-actions** (2026-09-21T15:47:23Z):

✅ Triggered [Buildkite CI #90244](https://buildkite.com/vllm/ci/builds/90244) for commit `9862c9be4c3a`.

**TheEpicDolphin** (2026-09-22T06:16:44Z):

@qiching the test failures seem related. Plz take a look when you get the chance

**qiching** (2026-09-22T17:53:41Z):

/ci run

**github-actions** (2026-09-22T17:54:04Z):

✅ Triggered [Buildkite CI #90469](https://buildkite.com/vllm/ci/builds/90469) for commit `372a9182071e`.

**qiching** (2026-09-22T20:09:41Z):

/ci run

**github-actions** (2026-09-22T20:10:03Z):

❌ This PR is 8 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5783346520 -->

**qiching** (2026-09-22T20:15:52Z):

/ci run

**github-actions** (2026-09-22T20:16:14Z):

✅ Triggered [Buildkite CI #90507](https://buildkite.com/vllm/ci/builds/90507) for commit `729132de6459`.

**qiching** (2026-09-22T21:55:23Z):

> @qiching the test failures seem related. Plz take a look when you get the chance

The related test failures are fixed. The remaining ones are unrelated: the kernels MoE IPC test is broken on main by [#57312](https://github.com/aarnphm/aarnphm.github.io/issues/57312) (fix in [#58107](https://github.com/aarnphm/aarnphm.github.io/issues/58107)), and the H200 MIG failures came from a network outage that broke HF model downloads (Network is unreachable / DNS errors).

**benchislett** (2026-09-22T21:56:20Z):

/ci retry

**github-actions** (2026-09-22T21:56:37Z):

✅ Queued 7 failed job(s) for retry in [Buildkite CI #90507](https://buildkite.com/vllm/ci/builds/90507).

**njhill** (2026-09-23T19:31:24Z):

@qiching @TheEpicDolphin @benchislett I found that this introduced a regression, please see [vllm-project/vllm#58434](https://github.com/vllm-project/vllm/pull/58434).

**qiching** (2026-09-24T04:17:14Z):

> @qiching @TheEpicDolphin @benchislett I found that this introduced a regression, please see [#58434](https://github.com/aarnphm/aarnphm.github.io/issues/58434).

Thanks @njhill, my oversight on the padded prompt tail case. Approved [#58434](https://github.com/aarnphm/aarnphm.github.io/issues/58434).

