---
title: '54255'
pageLayout: default
slug: thoughts/inference/vllm-prs/54255
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/54255.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#54255](https://github.com/aarnphm/aarnphm.github.io/issues/54255) \[Kimi-K3\] Add FlashInfer speculative KDA backend

[vllm-project/vllm#54255](https://github.com/vllm-project/vllm/pull/54255) · djmmoss · MERGED · created 2026-08-28T19:17:50Z · merged 2026-10-01T02:28:51Z

## Purpose

Integrate FlashInfer’s packed fused KDA kernel into Kimi-K3 speculative decode.
For supported pure speculative-decode batches, this replaces separate short
convolution, recurrent KDA, and gated normalization launches with
`flashinfer.packed_fused_kda_decode`. Unsupported configurations and mixed
batches retain the existing vLLM paths.

The backend defaults to `auto` and can be forced with
`additional_config.kda_spec_decode_backend={native,flashinfer}`. Eligibility is
fail-closed: the FlashInfer API, SM100-family CUDA, supported Kimi-K3 shapes and
dtypes, a bounded gate, the SD convolution-cache layout, and RecoverSSM being
disabled are all required. The recurrent state may be float32 or bfloat16;
bfloat16 needs [flashinfer-ai/flashinfer#5503](https://github.com/flashinfer-ai/flashinfer/issues/5503).

No competing open PR implementing this packed FlashInfer speculative-KDA path
was found. [#53525](https://github.com/aarnphm/aarnphm.github.io/issues/53525) optimizes the native single-token PDL path and does not
overlap this integration.

AI assistance was used in developing this contribution.

## Test Plan



```text
pytest -q tests/models/kimi_k3/test_kda.py \
  tests/models/kimi_k3/test_kda_metadata.py
```

The tests cover backend selection and fallback, cache-layout rejection, the
production `_forward` integration for float32 and bfloat16 recurrent state,
cache/state parity with the composed reference, null speculative rows,
accepted-token updates, and dynamic CUDA Graph metadata.

## Test Result

On one NVIDIA B300 (SM103), vLLM `2d50faf829` with [flashinfer-ai/flashinfer#5503](https://github.com/flashinfer-ai/flashinfer/issues/5503)
(`1e381ec10`):



```text
111 passed, 7 skipped
```

The 7 skips are existing single-token FlashInfer decode cases (DS conv layout,
unfused output norm).

### Real-model evaluation

The native and FlashInfer backends were evaluated on eight NVIDIA B300 GPUs
with `moonshotai/Kimi-K3` revision `a590ce090c` and the
`Inferact/Kimi-K3-DSpark` draft model revision `cf6b824462` using six draft
tokens. Both runs used vLLM head `3324e3c7e3`, FlashInfer head `609ec78b49`,
temperature 0, and seed 42.

| Evaluation                    |           Native |       FlashInfer | Result                                           |
| ----------------------------- | ---------------: | ---------------: | ------------------------------------------------ |
| GSM8K 5-shot, 1,319 questions | 94.693%, 153.1 s | 95.299%, 142.2 s | +0.607 percentage points; 1.077x lower wall time |
| OCRBench, 1,000 questions     |      908 and 905 |              906 | FlashInfer is within the native repeat range     |

For GSM8K, 1,283/1,319 extracted answers matched between backends. A full
FlashInfer repeat produced the exact same response checksum and completed in
141.1 s. OCRBench’s native responses and score varied across two concurrent
runs, so its latency is not used to claim a backend speedup; FlashInfer’s score
falls inside the observed native 905-908 range.

### Synthetic serving performance

An eight-B300 synthetic Kimi-K3/DSpark A/B (six draft tokens, 8K input, 1K
output) matched the baseline checksum. Total output throughput versus the
native backend:

| Recurrent state                               |    c=1 |    c=4 |    c=8 |   c=16 |   c=32 |
| --------------------------------------------- | -----: | -----: | -----: | -----: | -----: |
| float32 (default)                             | +2.63% | +2.54% | +3.29% | +1.76% | +1.75% |
| bfloat16 (`--mamba-ssm-cache-dtype bfloat16`) | +5.55% | +4.45% | +2.98% | +0.98% | +1.34% |

The bfloat16 row used this branch at `2d50faf829` with
[flashinfer-ai/flashinfer#5503](https://github.com/flashinfer-ai/flashinfer/issues/5503); both backends ran with bfloat16 state and
produced the same checksum.

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [x] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [x] The test plan, such as providing test command.
- [x] The test results, such as pasting the results comparison before and after, or e2e results
- [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**ywang96** (2026-08-30T11:21:32Z):

/ci run

**github-actions** (2026-08-30T11:21:39Z):

✅ @djmmoss, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**github-actions** (2026-08-30T11:21:50Z):

✅ Triggered [Buildkite CI #86221](https://buildkite.com/vllm/ci/builds/86221) for commit `6a5f4f82b0ae`.

**mergify** (2026-09-08T06:04:01Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @djmmoss.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**djmmoss** (2026-09-10T16:08:54Z):

@gau-nernst added the e2e results to the description. On 8x B300, GSM8K was 94.693% native vs 95.299% FlashInfer, and OCRBench was 908/905 across two native runs vs 906 FlashInfer.

**gau-nernst** (2026-09-17T05:12:31Z):

Hi @djmmoss, does this overlap with [#55364](https://github.com/aarnphm/aarnphm.github.io/issues/55364)?

**djmmoss** (2026-09-23T17:31:44Z):

not really, [#55364](https://github.com/aarnphm/aarnphm.github.io/issues/55364) covers the non-spec path (BF16 state, FlashInfer prefill and decode backends). this one adds the spec-decode path (T>1 verification with per-token checkpoints) through `packed_fused_kda_decode` from [flashinfer-ai/flashinfer#4809](https://github.com/flashinfer-ai/flashinfer/issues/4809), which just merged. it’s gated on that API, so `auto` stays on native until the next FI release.

**djmmoss** (2026-09-23T17:31:45Z):

/ci run

**github-actions** (2026-09-23T17:32:06Z):

❌ This PR is 755 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5799671721 -->

**djmmoss** (2026-09-23T17:47:07Z):

/ci run

**github-actions** (2026-09-23T17:47:28Z):

✅ Triggered [Buildkite CI #90725](https://buildkite.com/vllm/ci/builds/90725) for commit `d7e0972f6772`.

**djmmoss** (2026-09-28T16:50:43Z):

@gau-nernst ready for review, I added bf16 state support to flashinfer and updated here as well

**mergify** (2026-09-30T07:02:00Z):

Hi @djmmoss, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**djmmoss** (2026-09-30T16:40:42Z):

dropped the capability tests that just restated the gate or passed trivially off SM100, and kept the resolve test since it covers the auto/native/flashinfer selection and the fail-closed error. on naming, `packed_fused_kda_decode` is FlashInfer’s API name (it also exports `fused_kda_decode` and `packed_kda_decode`), so I kept it to match upstream and added a comment at the call site that packed is the varlen T>1 spec path.

also fixed the spec test after the main merge, which renamed the reference import to `fused_recurrent_kda_nvidia`.

**djmmoss** (2026-09-30T22:19:44Z):

/ci run

**github-actions** (2026-09-30T22:20:04Z):

❌ This PR is 14 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5920721682 -->

**gau-nernst** (2026-10-01T00:59:01Z):

/ci run

**github-actions** (2026-10-01T00:59:21Z):

✅ Triggered [Buildkite CI #92299](https://buildkite.com/vllm/ci/builds/92299) for commit `3ddbbee3d724`.

