---
title: '55747'
pageLayout: default
slug: thoughts/inference/vllm-prs/55747
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/55747.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#55747](https://github.com/aarnphm/aarnphm.github.io/issues/55747) \[Kimi Bug\] Fix kimi k3 AssertionError assert 0 <span>&lArr;</span> checkpoint_idx < len(blocks)

[vllm-project/vllm#55747](https://github.com/vllm-project/vllm/pull/55747) · yewentao256 · MERGED · created 2026-09-07T14:47:04Z · merged 2026-09-08T02:34:49Z

## Purpose



```bash

vllm serve moonshotai/Kimi-K3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --load-format fastsafetensors \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --reasoning-parser kimi_k3 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 30000 \
  --speculative-config '{"model":"RedHatAI/Kimi-K3-speculator.dspark","method":"dspark","num_speculative_tokens":8,"draft_sample_method":"probabilistic","rejection_sample_method":"standard"}'   --use-replayssm


HF_HUB_CACHE=/data/engine/hub_cache HF_HUB_OFFLINE=1 guidellm run   --backend '{"kind":"openai_http","target":"http://127.0.0.1:30000","model":"moonshotai/Kimi-K3","request_format":"/v1/chat/completions","timeout":100000,"extras":{"body":{"reasoning_effort":"max","temperature":1.0,"top_p":0.95}}}'   --tokenizer '{"kind":"huggingface_auto","model":"moonshotai/Kimi-K3","load_kwargs":{"trust_remote_code":true,"local_files_only":true}}'   --data 'kind=synthetic_text,prompt_tokens=8000,output_tokens=1000'   --profile 'kind=concurrent,warmup=0.1,cooldown=0.1'   --override profile.streams '1,4,16'   --constraint 'kind=max_duration,seconds=150'   --constraint 'kind=max_errors,count=10'   --seed 'kind=static,value=42'   --metrics 'kind=generative,sample_size=20'   --output 'kind=json,path=/home/yewentao256/guidellm-results/output_vllm_kimik3_8k1k.json'
```

This will raise error



```bash
09-06 15:49:23 [core.py:1387]     self.coordinator.cache_blocks(request, num_tokens_to_cache)
(EngineCore pid=862670) ERROR 09-06 15:49:23 [core.py:1387]   File "/home/yewentao256/vllm-source/vllm/v1/core/kv_cache_coordinator.py", line 778, in cache_blocks
(EngineCore pid=862670) ERROR 09-06 15:49:23 [core.py:1387]     manager.cache_blocks(
(EngineCore pid=862670) ERROR 09-06 15:49:23 [core.py:1387]   File "/home/yewentao256/vllm-source/vllm/v1/core/single_type_kv_cache_manager.py", line 1928, in cache_blocks
(EngineCore pid=862670) ERROR 09-06 15:49:23 [core.py:1387]     partial_hash = self._cache_partial_tail_block(request, num_tokens)
(EngineCore pid=862670) ERROR 09-06 15:49:23 [core.py:1387]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=862670) ERROR 09-06 15:49:23 [core.py:1387]   File "/home/yewentao256/vllm-source/vllm/v1/core/single_type_kv_cache_manager.py", line 1971, in _cache_partial_tail_block
(EngineCore pid=862670) ERROR 09-06 15:49:23 [core.py:1387]     assert 0 <= checkpoint_idx < len(blocks)
(EngineCore pid=862670) ERROR 09-06 15:49:23 [core.py:1387]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=862670) ERROR 09-06 15:49:23 [core.py:1387] AssertionError
```

This PR fixes the issue by saving additional checkpoint_idx to dict instead of recomputation (as the tokens might change by padding)

## Test

Now



```bash
(APIServer pid=957250) INFO 09-06 16:13:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.80, Accepted throughput: 495.45 tokens/s, Drafted throughput: 2196.56 tokens/s, Accepted: 4955 tokens, Drafted: 21968 tokens, Per-position acceptance rate: 0.685, 0.442, 0.291, 0.158, 0.101, 0.057, 0.040, 0.030, Avg Draft acceptance rate: 22.6%
(APIServer pid=957250)
```

---

## comments

**yewentao256** (2026-09-07T14:47:16Z):

/ci run

**github-actions** (2026-09-07T14:47:52Z):

✅ Triggered [Buildkite CI #87567](https://buildkite.com/vllm/ci/builds/87567) for commit `7d5e1a94ddde`.

**coderabbitai** (2026-09-07T14:48:02Z):

<!-- This is an auto-generated comment: summarize by coderabbit.ai -->

<!-- review_stack_entry_start -->

[![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/vllm-project/vllm/pull/55747)

<!-- review_stack_entry_end -->

<!-- recent_review_start -->

No actionable comments were generated in the recent review. 🎉

<details>
<summary>ℹ️ Recent review info</summary>

<details>
<summary>⚙️ Run configuration</summary>

**Configuration used**: Repository UI

**Review profile**: CHILL

**Plan**: Team

**Run ID**: `6c1c3d32-7187-46f8-976b-9f2110f433d3`

</details>

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between [`70584f6`](https://github.com/aarnphm/aarnphm.github.io/commit/70584f69b1bf866224604029fae038864a368019) and [`7d5e1a9`](https://github.com/aarnphm/aarnphm.github.io/commit/7d5e1a94ddde3bb0397b921b440c372a9b85cd0a).

</details>

<details>
<summary>📒 Files selected for processing (2)</summary>

- `tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py`
- `vllm/v1/core/single_type_kv_cache_manager.py`

</details>

**Included review availability:** Your plan provides up to 10 included reviews per hour; 9 remain after this review.

</details>

---

<!-- recent_review_end -->

<!-- walkthrough_start -->

<details>
<summary>📝 Summary</summary>

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **Bug Fixes**
  - Improved partial prefix-cache handling for hybrid attention and Mamba models.
  - Corrected checkpoint tracking during block allocation and cleanup.
  - Ensured cached tail blocks use the appropriate stored checkpoint information.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Walkthrough

The Mamba align cache manager now stores checkpoint positions and reserved block indexes in `_checkpoints`. Allocation, cleanup, partial-tail caching, and the related test use this unified state.

### Changes

**Mamba checkpoint bookkeeping**

| Layer / File(s)                                                                                                                                          | Summary                                                                                                                          |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| **Unified checkpoint state** <br> `vllm/v1/core/single_type_kv_cache_manager.py`                                                                         | The manager replaces `_checkpoint_positions` with `_checkpoints`, which stores the checkpoint position and reserved block index. |
| **Checkpoint lifecycle integration** <br> `vllm/v1/core/single_type_kv_cache_manager.py`, `tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py` | Allocation, cleanup, partial-tail caching, and the partial prefix-cache test use `_checkpoints` membership and values.           |

**Estimated code review effort:** 2 (Simple) | \~10 minutes

<!-- final_review_risk_start -->

**Merge Risk:** _⚪ Minimal_ · up to `7d5e1`

<!-- final_review_risk_coverage:{"sourceCommitId":"7d5e1a94ddde3bb0397b921b440c372a9b85cd0a","coveredCommitId":"7d5e1a94ddde3bb0397b921b440c372a9b85cd0a","kind":"reviewed"} -->

This change preserves the checkpoint block index selected before padding, preventing the speculative-decoding prefix-cache assertion without leaving an identified merge-blocking risk.

<!-- final_review_risk_end -->

**Suggested reviewers:** `dao007forever`, `zjy0516`

</details>

<!-- walkthrough_end -->

<!-- pre_merge_checks_walkthrough_start -->

<details>
<summary>🚥 Pre-merge checks | ✅ 4 | ❌ 1</summary>

### ❌ Failed checks (1 warning)

|     Check name     | Status     | Explanation                                                                                                                                                                               | Resolution                                                                         |
| :----------------: | :--------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------- |
| Docstring Coverage | ⚠️ Warning | Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 2 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. |

<details>
<summary>✅ Passed checks (4 passed)</summary>

|         Check name         | Status   | Explanation                                                                                                                    |
| :------------------------: | :------- | :----------------------------------------------------------------------------------------------------------------------------- |
|         Title check        | ✅ Passed | The title clearly identifies the Kimi K3 assertion failure and the checkpoint index condition that the pull request fixes.     |
|      Description check     | ✅ Passed | The description directly explains the failure, the padding-related cause, the checkpoint index fix, and the validation result. |
|     Linked Issues check    | ✅ Passed | Check skipped because no linked issues were found for this pull request.                                                       |
| Out of Scope Changes check | ✅ Passed | Check skipped because no linked issues were found for this pull request.                                                       |

</details>

</details>

<!-- pre_merge_checks_walkthrough_end -->

- [ ] <!-- {"checkboxId":"585bb3f6-faf5-4dbf-96d2-74e382adf19a"} --> Fix all pre-merge checks with AI

<!-- finishing_touch_checkbox_start -->

<details>
<summary>✨ Finishing Touches 💡 1</summary>

<!-- finishing_touch_suggestion:docstrings -->

<details>
<summary>📝 Generate docstrings 💡</summary>

- [ ] <!-- {"checkboxId":"7962f53c-55bc-4827-bfbf-6a18da830691"} --> Create stacked PR
- [ ] <!-- {"checkboxId":"3e1879ae-f29b-4d0d-8e06-d12b7ba33d98"} --> Commit on current branch

</details>

</details>

<!-- finishing_touch_checkbox_end -->

<!-- tips_start -->

---

Thanks for using [CodeRabbit](https://coderabbit.ai?utm_source=oss\&utm_medium=github\&utm_campaign=vllm-project/vllm\&utm_content=55747)! It’s free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

<details>
<summary>❤️ Share</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A\&url=https%3A//coderabbit.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai)
- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit\&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai\&mini=true\&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit\&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

</details>

<sub>Comment `@coderabbitai help` to get the list of available commands.</sub>

<!-- tips_end -->

**yewentao256** (2026-09-07T22:25:32Z):

/ci run

**github-actions** (2026-09-07T22:25:49Z):

✅ Triggered [Buildkite CI #87588](https://buildkite.com/vllm/ci/builds/87588) for commit `ea981d27b52b`.

