---
title: '59229'
pageLayout: default
slug: thoughts/inference/vllm-prs/59229
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59229.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59229](https://github.com/aarnphm/aarnphm.github.io/issues/59229) \[CI\]\[Kimi-K3\] Test prefix cache reuse with KV offload, P/D and DCP

[vllm-project/vllm#59229](https://github.com/vllm-project/vllm/pull/59229) · ZJY0516 · MERGED · created 2026-09-29T14:03:53Z · merged 2026-10-02T12:58:36Z

## Purpose

Kimi-K3 has no end-to-end prefix-cache coverage. FlashKDA exports Mamba prefill checkpoints, a path the Qwen3.5-based `test_hybrid_prefix_cache_hit_rate.py` never takes, so regressions such as the dropped prompt-end checkpoint fixed by [#59146](https://github.com/aarnphm/aarnphm.github.io/issues/59146) / [#59175](https://github.com/aarnphm/aarnphm.github.io/issues/59175) reach users unnoticed. KV offload, P/D and DCP are not tested with K3 at all.

This adds `tests/models/kimi_k3/test_prefix_cache.py`. It serves [`riverclouds/Kimi-K3-8L-dummy`](https://huggingface.co/riverclouds/Kimi-K3-8L-dummy) (the official Kimi-K3 config with 8 layers and no weights) with `--load-format dummy`, and runs 3 turns where each turn extends the previous prompt. Each turn must:

- hit at least the previous prompt’s end checkpoint (`cached_tokens`), and
- produce the same greedy tokens as a recompute of the same prompt under a fresh `cache_salt` (`check_logprobs_close`). This catches restored state that is wrong even though the hit count looks right.

With offloading, GPU prefix caches are reset before each turn, so the hit has to come back from CPU.

Every deployment runs in 4 modes:

- `block`: Mamba-block granularity.
- `partial`: `--prefix-match-unit 128`, i.e. partial hits.
- `block-dspark` and `partial-dspark`: the same, plus DSpark speculative decoding as in the recipe. The draft is [`riverclouds/Kimi-K3-DSpark-for-8L-dummy`](https://huggingface.co/riverclouds/Kimi-K3-DSpark-for-8L-dummy), which is `Inferact/Kimi-K3-DSpark` with `target_layer_ids` remapped to the 8-layer target and no weights. With EAGLE-style block drop, the expected hit is one unit lower.

| deployment                                                                                                                               | GPUs |
| ---------------------------------------------------------------------------------------------------------------------------------------- | ---- |
| plain                                                                                                                                    | 1    |
| SimpleCPUOffloadConnector (config from the [Kimi-K3 recipe](https://recipes.vllm.ai/moonshotai/Kimi-K3?kv_offload=simple))               | 1    |
| DCP2                                                                                                                                     | 2    |
| DCP2 + SimpleCPUOffloadConnector                                                                                                         | 2    |
| P/D (NIXL) TP1 → TP1                                                                                                                     | 2    |
| P/D TP2 → DEP2 (decode: DP2 + EP, TP1 per rank; prefix caching on P only, since NIXL rejects it on a hybrid decoder with a different TP) | 4    |
| P/D + SimpleCPUOffloadConnector (MultiConnector)                                                                                         | 2    |

CI: a new `(4xB200) Kimi K3 Prefix Cache Shard %N` step runs the file in 2 shards. It installs NIXL with `.buildkite/scripts/install-kv-connectors.sh` first, because the CI image does not include it. It is triggered by `vllm/v1/core/`, `vllm/v1/spec_decode/`, the NIXL and SimpleCPUOffload connectors, `vllm/model_executor/layers/mamba/` and `vllm/models/kimi_k3/`. The existing 1xB200 Kimi K3 step ignores this file.

### Known failures (strict xfail)

- `dcp2-*-dspark`, `dcp2-offload-*-dspark`: FlashInfer MLA DCP decode crashes on ragged spec-decode batches (`causal_seqlens_kv_global must have shape (9,), got (2,)`). This is fixed by [#59392](https://github.com/aarnphm/aarnphm.github.io/issues/59392).

### Not covered

P/D + DCP: NIXL with DCP sets `cp_kv_cache_interleave_size` to the block size (`vllm/config/vllm.py`), but FlashInfer MLA native DCP requires 1, so the server fails to start. A P TP → D DCP case can be added once that is fixed.

### Depends on [#59146](https://github.com/aarnphm/aarnphm.github.io/issues/59146) or [#59175](https://github.com/aarnphm/aarnphm.github.io/issues/59175)

On current `main`, every block-granularity case fails at turn 1 with `cached=0 expected=17280`, which is the bug those PRs fix. Partial cases pass on `main`. This PR stays a draft until one of them lands.

### Not a duplicate

There are no open PRs adding Kimi-K3 prefix-cache, offload, P/D or DCP tests. [#59146](https://github.com/aarnphm/aarnphm.github.io/issues/59146) and [#59175](https://github.com/aarnphm/aarnphm.github.io/issues/59175) fix the scheduler bug and add unit tests only.

## Test Plan

On GB200 (SM100), with [#59146](https://github.com/aarnphm/aarnphm.github.io/issues/59146) cherry-picked onto main (`379e9a1`):



```
pytest -v -s tests/models/kimi_k3/test_prefix_cache.py
pre-commit run --files tests/models/kimi_k3/test_prefix_cache.py .buildkite/test_areas/models_basic.yaml
```

## Test Result

On main + [#59146](https://github.com/aarnphm/aarnphm.github.io/issues/59146), one 4xGB200 node. The full run predates two changes: `pd-tp1-tp2` was replaced by `pd-tp2-dep2`, and `pd-dcp2` was dropped. It ran 32 cases: `24 passed, 4 failed, 4 xfailed in 5136.53s (1:25:36)`. The 4 failures are the `dcp2*-dspark` cases, marked xfail after the run started. The 4 xfails were `pd-dcp2`. The `pd-tp2-dep2` cases were run separately on this branch at `c2a0aaa` (merged with main): `4 passed in 1138.28s (0:18:58)`.

| deployment         | block | partial | block-dspark                                                                | partial-dspark                                                              |
| ------------------ | ----- | ------- | --------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| plain, offload     | pass  | pass    | pass                                                                        | pass                                                                        |
| dcp2, dcp2-offload | pass  | pass    | xfail ([#59392](https://github.com/aarnphm/aarnphm.github.io/issues/59392)) | xfail ([#59392](https://github.com/aarnphm/aarnphm.github.io/issues/59392)) |
| pd, pd-offload     | pass  | pass    | pass                                                                        | pass                                                                        |
| pd-tp2-dep2        | pass  | pass    | pass                                                                        | pass                                                                        |

Each case prints its turns. For example, `dcp2-offload-partial`, with GPU prefix caches reset before every turn:



```
turn  prompt  cached  expected  matches_recompute
   0    8869       0         0  True
   1    9646    8832      8832  True
   2   10423    9600      9600  True
```

With DSpark (`pd-offload-partial-dspark`), the expected hit drops by one 128-token unit:



```
turn  prompt  cached  expected  matches_recompute
   0   18469       0         0  True
   1   19246   18304     18304  True
   2   20023   19072     19072  True
```

On main without the fix, `plain-block` fails at turn 1 (`cached=0 expected=17280`) and `plain-partial` passes. Each server start takes about 70–90 s.

With `OffloadingConnector` instead, DCP2 hits CPU only when the prompt-end checkpoint lands on a multiple of the DCP-scaled MLA chunk. That connector is not in the recipe, so it is not covered here.

No model outputs change (test and CI only), so no eval was run.

AI assistance (Claude Code) was used to write this test.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**ZJY0516** (2026-09-30T13:04:11Z):

/ci run —allow-stale

**github-actions** (2026-09-30T13:04:32Z):

✅ Triggered [Buildkite CI #92145](https://buildkite.com/vllm/ci/builds/92145) for commit `947b2bbf64c0`.

⚠️ This PR is 89 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

**mergify** (2026-09-30T13:16:16Z):

Hi @ZJY0516, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-09-30T13:56:25Z):

Hi @ZJY0516, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**ZJY0516** (2026-09-30T14:06:59Z):

/ci run —allow-stale

**github-actions** (2026-09-30T14:07:23Z):

✅ Triggered [Buildkite CI #92163](https://buildkite.com/vllm/ci/builds/92163) for commit `9293b44729d0`.

⚠️ This PR is 1 commit behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

**vllm-agent** (2026-09-30T15:16:46Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 10 test steps (23 jobs) instead of 10 (25 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 10 (25)       | 10 (23)  | 1 (3)      | 1 (1)     |
| AMD mirrors            | 8 (23)        | 9 (24)   | 0 (0)      | 1 (1)     |

<details><summary>Selector would run (10)</summary>

- `basic-models-test-other-cpu`
- `basic-models-tests-extra-initialization` ×14
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `deepseek-v4-kernel-test-b200`
- `glm5next-unit-tests`
- `inkling-unit-tests-b200`
- `kimi-k3-unit-tests-b200`
- `qwen4-exp-unit-tests`
- `qwen4-exp-unit-tests-cpu`

</details>

<details><summary>Would skip (today's rules run them) (1)</summary>

- `distributed-model-tests-2-gpus` ×3

</details>

<details><summary>Would add (today's rules do not run them) (1)</summary>

- `deepseek-v4-kernel-test-b200` (code map)

</details>

<details><summary>AMD mirrors: would skip (0)</summary>

none

</details>

<details><summary>AMD mirrors: would add (1)</summary>

- `deepseek-v4-kernel-test-b200` (code map)

</details>

<sub>2 changed files · base `b558f160a2` · head `774cf86a7f` · Python record: build 92561 at `4056c8ac1f` · kernel record: table [`4056c8a`](https://github.com/aarnphm/aarnphm.github.io/commit/4056c8ac1f) (build 92561), map [`4056c8a`](https://github.com/aarnphm/aarnphm.github.io/commit/4056c8ac1f) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 0 optional steps the selector would also run</sub>

**ivanium** (2026-10-01T21:54:57Z):

sorry for being late. Overall great, one minor comment is how about we do PD TP2<span>&rarr;</span>DEP2? which is perhaps closer to the real use case.

**ZJY0516** (2026-10-02T08:40:36Z):

> I know that kimi is fully supported by AMD and that we do have the necessary kernels. And since we are trying to close the parity gap in ci, could you add rocm support on this test?

You can update it after I merge this

**ZJY0516** (2026-10-02T08:41:05Z):

/ci run —allow-stale

**github-actions** (2026-10-02T08:41:26Z):

✅ Triggered [Buildkite CI #92584](https://buildkite.com/vllm/ci/builds/92584) for commit `774cf86a7f91`.

⚠️ This PR is 2 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

