---
title: '56794'
pageLayout: default
slug: thoughts/inference/vllm-prs/56794
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56794.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56794](https://github.com/aarnphm/aarnphm.github.io/issues/56794) \[Bugfix\]\[Kimi-K3\] Keep transient checkpoints out of prefix-cache eviction

[vllm-project/vllm#56794](https://github.com/vllm-project/vllm/pull/56794) · Sy0307 · MERGED · created 2026-09-14T08:57:34Z · merged 2026-09-15T06:18:25Z

## Summary

Keep transient KDA/Mamba prefill checkpoints out of the ordinary prefix-cache eviction set when sparse retention is explicitly active.

This change targets Kimi-K3’s hybrid attention path.

## Problem

FlashKDA creates internal checkpoints so the current prefill can continue from a recurrent-state boundary without an additional scheduling step. A checkpoint in the middle of a prompt is useful to the current request, but it is not necessarily a reusable shared-prefix boundary.

Before this change, such a checkpoint could be re-keyed and registered as a normal prefix-cache block. It then participated in the same GPU LRU eviction pool as reusable hot prefixes:



```text
middle checkpoint -> hash -> prefix-cache LRU -> evicts hot prefix
                                             -> later local hit becomes a miss
                                             -> extra load or prefill work
```

Before/after cache lifecycle:



```text
Before:

  KDA prefill
      |
      +--> temporary checkpoint
      |        |
      |        +--> hash + ordinary LRU
      |                     |
      |                     +--> competes with hot prefix
      |                                  |
      |                                  +--> local prefix miss

After:

  KDA prefill
      |
      +--> temporary checkpoint ------------------> current request can use it
      |                                              |
      |                                              +--> no ordinary LRU entry
      |
      +--> shared-prefix-boundary checkpoint ------> reusable prefix cache
                                                     |
                                                     +--> may be retained/shared
```

This is particularly problematic when `prefix_cache_retention_interval == 0`: intermediate checkpoints are not intended to be retained as dense semantic checkpoints, but their temporary hash could still make them eviction candidates.

## Fix

Pass the active retention interval into checkpoint publication. When sparse retention is active and the checkpoint is neither at the end of the prompt nor at the shared-prefix boundary, treat it as request-local state:



```text
middle checkpoint -> current request may use it
                  -> temporary hash is removed
                  -> no ordinary prefix-cache LRU entry
                  -> reusable hot prefixes remain protected
```

Prompt-end and shared-prefix-boundary checkpoints retain the existing publication behavior. This change does not disable KDA checkpoint computation or change the state written by FlashKDA.

## Performance

On a representative B200 Kimi-K3 hybrid workload using TP8/PP2/direct DCP8, the isolated production-style comparison was:

| Configuration                                              | Output throughput |
| ---------------------------------------------------------- | ----------------: |
| Checkpoint disabled (diagnostic baseline)                  |  1013.00 tokens/s |
| Checkpoint enabled with transient-checkpoint retention fix |  1091.80 tokens/s |

The fixed configuration improved throughput by 7.78% over the checkpoint-disabled diagnostic baseline while retaining the internal checkpoint feature. A separate run with the GEMM-AR option disabled measured 1108.89 tokens/s; that result is included only as context and is not an isolated measurement of this patch.

These measurements are workload-specific and are not used as a claim that this small cache-lifecycle change explains every difference between software stacks. The primary regression signal is the cache-manager reproduction: temporary checkpoint publication changes which blocks compete in the local prefix-cache eviction pool.

## Why this is not a duplicate

- PR [#55760](https://github.com/aarnphm/aarnphm.github.io/issues/55760) changes the default sparse/dense retention policy for Mamba + EAGLE; this change keeps the explicit sparse-retention behavior but fixes transient checkpoint publication.
- PR [#56615](https://github.com/aarnphm/aarnphm.github.io/issues/56615) fixes MooncakeStore boundary pairing between Mamba/KDA and attention KV; this change is limited to local prefix-cache block retention.
- PR [#56525](https://github.com/aarnphm/aarnphm.github.io/issues/56525) guards cache hits until the writing scheduler step is committed; this change addresses eviction eligibility, not write visibility.
- PR [#52527](https://github.com/aarnphm/aarnphm.github.io/issues/52527) adds observability for sparse-retention misses; this change modifies the cache lifecycle.

## Validation

Focused prefix-cache tests should be run with:



```bash
pytest tests/v1/core/test_prefix_caching.py -q
```

The change is intentionally small and preserves the current-request checkpoint path. Model-scale performance validation is still required on a representative hybrid KDA workload.

This change was prepared with AI assistance; the author should review the complete diff and test results before merging.

---

## comments

**ZJY0516** (2026-09-14T12:27:19Z):

I directly pushed some changes:

- Made retention_interval a required parameter of \_cache_partial_tail_block so future callers can’t silently fall back to dense behavior
- Replaced the `block_pool.evict_blocks(...)` with a direct `block_pool._maybe_evict_cached_block(checkpoint_block)` call
- Added regression tests

**ZJY0516** (2026-09-14T12:30:21Z):

/ci run

**github-actions** (2026-09-14T12:30:38Z):

✅ Triggered [Buildkite CI #88813](https://buildkite.com/vllm/ci/builds/88813) for commit `b1bca7b11dc5`.

**Sy0307** (2026-09-14T20:14:16Z):

Follow-up: updated the existing Mamba checkpoint regression test to pass the newly required `retention_interval` argument. The earlier CI failure was a stale direct test call; no production logic changed in this fix.

**Sy0307** (2026-09-14T20:14:17Z):

/ci run

**github-actions** (2026-09-14T20:14:42Z):

✅ Triggered [Buildkite CI #88907](https://buildkite.com/vllm/ci/builds/88907) for commit `be1a1c940da9`.

**Sy0307** (2026-09-15T03:28:48Z):

CI have passed. Can you merge it? Thx. @ZJY0516

