---
title: KV cache layouts
pageLayout: default
slug: thoughts/inference/kv-layout
permalink: https://aarnphm.xyz/thoughts/inference/kv-layout.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# hetero TP in P/D disagg + the KV layout fight (BLHNC vs LBHNC)

_the whole debate reduces to one question: which axis do you make contiguous, and who pays for the axes you didn’t._

vllm main @ [`c059bee8`](https://github.com/vllm-project/vllm/tree/c059bee812842fb76c2b045e240c02f37c0d498d) (2026-10-11). every `file:line` below is at that commit. Slack was unavailable through the connector, so this is built from code, git history, the RFC, and PR review threads. claims marked _(inferred)_ are my reading of the code and haven’t been run.

For P/D transfer geometry, see <ref slug="kv-transfer">.

---

## 0. the vocabulary (RFC [#42082](https://github.com/aarnphm/aarnphm.github.io/issues/42082), Lucas Wilkinson, 2026-05-08)

[vllm#42082 “\[RFC\]: Standardize KV-cache Layouts”](https://github.com/vllm-project/vllm/issues/42082) defines the layout vocabulary. it’s open, assigned to Lucas, Nick Lucchesi, and Matt Bonanni.

one **logical** shape for every backend, every cache type:



```
[L, B, H, N, C]
 L = layer slot
 B = block
 H = heads            (1 for MLA / headless)
 N = states per block (block_size for attention, block_size/4 for DSV4 c4a, 1 for recurrent state)
 C = state content    (always contiguous; GQA: [2, head_dim], so K and V are packed per token)
```

physical layout = a stride permutation of that logical shape. the enum is at [`vllm/v1/kv_cache_layout.py:15`](https://github.com/vllm-project/vllm/blob/c059bee812842fb76c2b045e240c02f37c0d498d/vllm/v1/kv_cache_layout.py#L15):

| layout    | physical order | old name | what is contiguous                               | the RFC’s pitch                                 |
| --------- | -------------- | -------- | ------------------------------------------------ | ----------------------------------------------- |
| **LBHNC** | `[L,B,H,N,C]`  | HND      | one layer’s page; inside it, one head’s `[N,C]`  | hetero-TP friendly (head slices contiguous)     |
| LBNHC     | `[L,B,N,H,C]`  | NHD      | one layer’s page; inside it, one token’s `[H,C]` | hetero block size friendly                      |
| **BLHNC** | `[B,L,H,N,C]`  | none     | one **block across all layers**                  | one RDMA per block, homogeneous TP + block size |
| BHLNC     | `[B,H,L,N,C]`  | none     | one **(block, head) across all layers**          | big transfers _and_ hetero TP                   |
| LHBNC     | `[L,H,B,N,C]`  | none     | per-head pools (ROCm K/V group-first kernels)    | AITER / ROCm kernels                            |
| BLNHC     | `[B,L,N,H,C]`  | none     | block across layers, token-major inside          | completeness                                    |

the predicates on the enum are the actual decision rules everyone downstream keys off:

- `is_block_contiguous`: last three dims are `H,N,C`. true for LBHNC, BLHNC. **false for BHLNC.** NIXL gates hetero TP on this.
- `is_block_compact`: L and B are the two outermost dims. true for LBHNC, LBNHC, BLHNC, BLNHC. required when specs with different `(H,N,C)` alias the same bytes (hybrid models).
- `is_block_outermost`: B first. BLHNC, BLNHC, BHLNC. NIXL calls these “packed caches.”

legacy `NHD`/`HND` survive as aliases: `NHD→LBNHC`, `HND→LBHNC` ([`config/cache.py:21`](https://github.com/vllm-project/vllm/blob/c059bee812842fb76c2b045e240c02f37c0d498d/vllm/config/cache.py#L21)).

## 3. what each layout costs the kernels

the attention kernel sees one layer at a time: a 4D view `[B, H, N, C]` with whatever strides the layout implies (`layer_view_order`).

- **LBHNC**: FlashInfer’s native HND. on Blackwell, FlashInfer’s trtllm-gen kernels _only_ take head-major block interiors: `supported_kv_cache_layouts` returns `(LBHNC, BLHNC)` on SM100 ([`flashinfer.py:605`](https://github.com/vllm-project/vllm/blob/c059bee812842fb76c2b045e240c02f37c0d498d/vllm/v1/attention/backends/flashinfer.py#L605)). the comment says it straight: “the L/B nesting outside the block is immaterial to them.” FA handles HND through strides (see [#47314](https://github.com/aarnphm/aarnphm.github.io/issues/47314) for the packed-HND reshape bug).
- **BLHNC**: identical interior, but the block stride is `L×` bigger. every kernel that assumed `stride(0) == page_bytes` breaks: the TRT-LLM contiguity assertion ([#34158](https://github.com/aarnphm/aarnphm.github.io/issues/34158)), MLA kernels (disabled in [#37090](https://github.com/aarnphm/aarnphm.github.io/issues/37090), re-enabled via stride-aware kernels in [#45111](https://github.com/aarnphm/aarnphm.github.io/issues/45111), Yifan Qiao), per-token-head quant scales (cross-layer disabled in [#49226](https://github.com/aarnphm/aarnphm.github.io/issues/49226)), FlashInfer sparse MLA addressing ([#54465](https://github.com/aarnphm/aarnphm.github.io/issues/54465)). the per-step kernel cost is roughly neutral _(inferred)_: each page is still one contiguous run. you pay in kernel-compatibility surface area, not FLOPs.
- **BHLNC**: per-layer view has head stride `L·N·C`, so each head’s `[N,C]` tile in a layer is a separate run of `N × C × 2 B` (16 × 256 × 2 = 8 KiB). still coalescable. but backends must support an arbitrary head stride, and almost none declare BHLNC (FlashInfer maps it to “HND” in its name table, [`backends/utils.py:175`](https://github.com/vllm-project/vllm/blob/c059bee812842fb76c2b045e240c02f37c0d498d/vllm/v1/attention/backends/utils.py#L175), but SM100 doesn’t list it as supported).
- **K/V packing** (\[2/N\] [#44455](https://github.com/aarnphm/aarnphm.github.io/issues/44455)): C = `[2, head_dim]`, so one token’s K row and V row sit adjacent. this is what made “one descriptor per (layer, block)” possible at all; the old `(2, num_blocks, ...)` shape doubled descriptors by splitting K and V.

## 4. who decides the layout

single writer, once, in engine core: [`resolve_kv_cache_layout`, `backends/utils.py:259`](https://github.com/vllm-project/vllm/blob/c059bee812842fb76c2b045e240c02f37c0d498d/vllm/v1/attention/backends/utils.py#L259). priority:

1. already set on `cache_config` (tests / tools)
2. intersect every backend’s `supported_kv_cache_layouts()` (most preferred first)
3. narrow to `is_block_compact` if specs have mixed `(H, N, page_bytes)` (hybrid models)
4. narrow to `is_block_outermost` if DCP mixes a sharded target with a replicated draft at different page sizes
5. `VLLM_KV_CACHE_LAYOUT` must be a surviving candidate, or it’s a hard error
6. else the connector’s preference (NIXL says **LBHNC** for non-MLA), dropped with a warning if it’s illegal
7. else LBNHC

then it’s broadcast to workers via the `set_kv_cache_layout` RPC. P and D resolve **independently**, and nothing guarantees they agree until the NIXL handshake checks.

