---
title: '44454'
pageLayout: default
slug: thoughts/inference/vllm-prs/44454
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/44454.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#44454](https://github.com/aarnphm/aarnphm.github.io/issues/44454) \[1/N\]\[KV-Cache Layout Refactor\] Refactor DSV4 KV cache config construction

[vllm-project/vllm#44454](https://github.com/vllm-project/vllm/pull/44454) · LucasWilkinson · MERGED · created 2026-06-03T22:22:25Z · merged 2026-06-07T14:53:37Z

PR [#42374](https://github.com/aarnphm/aarnphm.github.io/issues/42374) (first part of RFC [#42082](https://github.com/aarnphm/aarnphm.github.io/issues/42082)) has been split into 4 PRs:

<span>&rarr;</span> [#44454](https://github.com/aarnphm/aarnphm.github.io/issues/44454) \[1/N\]\[KV-Cache Layout Refactor\] Refactor DSV4 KV cache config
[#44455](https://github.com/aarnphm/aarnphm.github.io/issues/44455) \[2/N\]\[KV-Cache Layout Refactor\] Pack K/V into the content dim across attention backends
[#44456](https://github.com/aarnphm/aarnphm.github.io/issues/44456) \[3/N\]\[KV-Cache Layout Refactor\] Standardize Mamba cache; drop `get_transfer_cache_regions`
[#44458](https://github.com/aarnphm/aarnphm.github.io/issues/44458) \[4/N\]\[KV-Cache Layout Refactor\] Standardize KV cache layout

## Summary

Extracts the per-page-size layer bucketing logic out of `_get_kv_cache_config_deepseek_v4` into a reusable helper, `_bucket_layers_by_page_size`. Behavior-preserving: same allocation pattern (one `KVCacheTensor` per `(slot_idx, page_size)`), same `shared_by` semantics.

## Changes

- Add `_bucket_layers_by_page_size(kv_cache_groups)` handling both `UniformTypeKVCacheSpecs` (per-layer page sizes) and regular specs.
- Rewrite `_get_kv_cache_config_deepseek_v4()` to use it.
- Simplify the DSV4 branch of `_pool_bytes_per_block()` to use bucket-derived values.

## Test Results (DeepSeek-V4-Flash, TP=4xB200, fp8 KV cache)

- KV cache: 333,437 tokens (+1.16% vs main’s 329,601) — previous code over-counted per-block cost when slot counts differ across page sizes
- GSM8K: 97.0% accuracy (5-shot, 200q, temp=0)

## AI assistance

This PR was prepared with AI assistance (Claude).

---

## comments

**mergify** (2026-06-04T04:03:05Z):

Hi @LucasWilkinson, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> failing?</summary>
> <br/>
> <code>mypy</code> is run differently in CI. If the failure is related to this check, please use the following command to run it locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> ```
>
> </details>

