---
title: '51718'
pageLayout: default
slug: thoughts/inference/vllm-prs/51718
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/51718.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#51718](https://github.com/aarnphm/aarnphm.github.io/issues/51718) \[6/N\]\[KV-Cache Layout Refactor\] Standardize KV cache layout

[vllm-project/vllm#51718](https://github.com/vllm-project/vllm/pull/51718) · LucasWilkinson · MERGED · created 2026-08-10T18:30:58Z · merged 2026-08-22T00:47:50Z

## Purpose

The core of the KV-cache layout standardization series (RFC [#42082](https://github.com/aarnphm/aarnphm.github.io/issues/42082)).

Standardizes every KV cache allocation on the logical `[L, B, H, N, C]` vocabulary:

- `KVCacheLayout` enumerates the physical stride permutations (`LBHNC`, `LBNHC`, `LHBNC`, `BLHNC`, `BLNHC`, `BHLNC`)
- Layout resolution has a single writer: attention-backend selection publishes the layout into `CacheConfig` (test override > backend-required > `VLLM_KV_CACHE_LAYOUT` > connector preference > `LBNHC`); every consumer reads it from there.
- `indexes_kv_by_block_stride` and per-backend cache-shape/stride-order hooks are removed — the standardized views make them constant.

Replaces [#44458](https://github.com/aarnphm/aarnphm.github.io/issues/44458) (same content, restacked).

## Test Plan / Result

FULL CI Run; No PR related failures

AI-assisted (Claude); reviewed by the
submitter.

---

## comments

**mergify** (2026-08-10T18:31:40Z):

Documentation preview: [https://vllm—51718.org.readthedocs.build/en/51718/](https://vllm--51718.org.readthedocs.build/en/51718/)

**mergify** (2026-08-10T18:39:02Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-08-11T15:20:20Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-08-12T06:43:48Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**LucasWilkinson** (2026-08-13T18:50:11Z):

/ci run

**github-actions** (2026-08-13T18:50:28Z):

✅ Triggered [Buildkite CI #83782](https://buildkite.com/vllm/ci/builds/83782) for commit `9d61ce258b5c`.

**LucasWilkinson** (2026-08-16T04:31:53Z):

/ci run

**github-actions** (2026-08-16T04:32:10Z):

✅ Triggered [Buildkite CI #84065](https://buildkite.com/vllm/ci/builds/84065) for commit `28da35b29eb0`.

**LucasWilkinson** (2026-08-16T14:15:05Z):

/ci retry

**github-actions** (2026-08-16T14:15:55Z):

✅ No failed, timed-out, or expired jobs need retrying: <https://buildkite.com/vllm/ci/builds/84065>

**LucasWilkinson** (2026-08-16T23:15:33Z):

/ci run

**github-actions** (2026-08-16T23:15:50Z):

✅ Triggered [Buildkite CI #84126](https://buildkite.com/vllm/ci/builds/84126) for commit `49e1e208c972`.

**LucasWilkinson** (2026-08-17T04:36:58Z):

/ci run

**github-actions** (2026-08-17T04:37:14Z):

✅ Triggered [Buildkite CI #84152](https://buildkite.com/vllm/ci/builds/84152) for commit `83c917d341a0`.

**mergify** (2026-08-17T11:51:23Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**LucasWilkinson** (2026-08-17T21:47:13Z):

/ci run

**github-actions** (2026-08-17T21:47:31Z):

✅ Triggered [Buildkite CI #84271](https://buildkite.com/vllm/ci/builds/84271) for commit `123512d18227`.

**LucasWilkinson** (2026-08-18T01:59:56Z):

/ci run

**github-actions** (2026-08-18T02:00:15Z):

✅ Triggered [Buildkite CI #84307](https://buildkite.com/vllm/ci/builds/84307) for commit `c612dac75c6b`.

**LucasWilkinson** (2026-08-18T19:01:09Z):

/ci run

**github-actions** (2026-08-18T19:01:30Z):

✅ Triggered [Buildkite CI #84452](https://buildkite.com/vllm/ci/builds/84452) for commit `ea43eed5adf4`.

**LucasWilkinson** (2026-08-18T20:55:37Z):

/ci run

**github-actions** (2026-08-18T20:56:01Z):

✅ Triggered [Buildkite CI #84472](https://buildkite.com/vllm/ci/builds/84472) for commit `de177f7d5469`.

**LucasWilkinson** (2026-08-18T23:42:13Z):

/ci run

**github-actions** (2026-08-18T23:42:31Z):

✅ Triggered [Buildkite CI #84501](https://buildkite.com/vllm/ci/builds/84501) for commit `1544e6023822`.

**mergify** (2026-08-19T04:34:32Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**LucasWilkinson** (2026-08-19T04:37:44Z):

/ci run

**github-actions** (2026-08-19T04:38:02Z):

✅ Triggered [Buildkite CI #84542](https://buildkite.com/vllm/ci/builds/84542) for commit `6cba88705710`.

**njhill** (2026-08-19T15:25:56Z):

There was temporary unrelated CI regression in main (my fault :( now fixed) which is the cause of two of the latest failures. I retried a couple of the other ones.

I think we might need to do `/ci run all` for the full suite.

**mergify** (2026-08-19T17:10:22Z):

Hi @LucasWilkinson, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-08-19T20:04:46Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**njhill** (2026-08-19T21:24:45Z):

I did high level pass review, nothing stands out!

**LucasWilkinson** (2026-08-19T21:30:18Z):

/ci run all

**github-actions** (2026-08-19T21:30:36Z):

✅ Triggered [Buildkite CI #84690](https://buildkite.com/vllm/ci/builds/84690) for commit `a36e77d3eb10`.

**LucasWilkinson** (2026-08-19T21:43:13Z):

/amd-ci run nightly

**github-actions** (2026-08-19T21:43:32Z):

✅ Triggered [Buildkite AMD CI #12253](https://buildkite.com/vllm/amd-ci/builds/12253) for commit `a36e77d3eb10`.

**LucasWilkinson** (2026-08-20T04:57:13Z):

/amd-ci run nightly

**LucasWilkinson** (2026-08-20T04:57:19Z):

/ci run all

**github-actions** (2026-08-20T04:57:31Z):

✅ Triggered [Buildkite AMD CI #12269](https://buildkite.com/vllm/amd-ci/builds/12269) for commit `3d52cd0f34cd`.

**github-actions** (2026-08-20T04:57:48Z):

✅ Triggered [Buildkite CI #84747](https://buildkite.com/vllm/ci/builds/84747) for commit `3d52cd0f34cd`.

**mergify** (2026-08-20T10:32:40Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-08-20T21:08:59Z):

Hi @LucasWilkinson, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**LucasWilkinson** (2026-08-20T21:15:45Z):

/amd-ci run nightly

**LucasWilkinson** (2026-08-20T21:15:45Z):

/ci run all

**github-actions** (2026-08-20T21:16:02Z):

✅ Triggered [Buildkite AMD CI #12288](https://buildkite.com/vllm/amd-ci/builds/12288) for commit `aa356da7bf0d`.

**github-actions** (2026-08-20T21:16:22Z):

✅ Triggered [Buildkite CI #84890](https://buildkite.com/vllm/ci/builds/84890) for commit `aa356da7bf0d`.

**LucasWilkinson** (2026-08-21T01:20:46Z):

/amd-ci run nightly

**LucasWilkinson** (2026-08-21T01:20:55Z):

/ci run all

**github-actions** (2026-08-21T01:21:04Z):

✅ Triggered [Buildkite AMD CI #12296](https://buildkite.com/vllm/amd-ci/builds/12296) for commit `4eae6fa27544`.

**github-actions** (2026-08-21T01:21:21Z):

✅ Triggered [Buildkite CI #84920](https://buildkite.com/vllm/ci/builds/84920) for commit `4eae6fa27544`.

**AndreasKaratzas** (2026-08-21T06:37:13Z):

We’ve been seeing a strange CrossLayer KV accuracy failure in AMD CI. In \[[build 12283](https://buildkite.com/vllm/amd-ci/builds/12283#01a0208f-90ca-4679-ad77-da1ebadb049c)\](<https://buildkite.com/vllm/amd-ci/builds/12283#01a0208f-90ca-4679-ad77-da1ebadb049c>), Qwen worked normally when the prefiller and decoder were both TP2, but TP1 prefiller <span>&rarr;</span> TP2 decoder completed every request successfully and still got exactly `0.0` accuracy.

It initially looked like an intermittent runner or dependency problem, but the same TP-specific failure reproduced elsewhere. My understanding is that the standardized layout caused regular BLHNC cache storage to take the packed-MLA registration path, which gives NIXL the wrong offsets when the P/D TP sizes differ. The change in `9b253bc6` keeps that packed path limited to MLA and appears to fix the actual corruption.

After that change, the same job passed all six Qwen and DeepSeek configurations in \[[build 12288](https://buildkite.com/vllm/amd-ci/builds/12288#01a02108-4e24-46f5-9595-d8b18a85b24e)\](<https://buildkite.com/vllm/amd-ci/builds/12288#01a02108-4e24-46f5-9595-d8b18a85b24e>). Could we keep this fix in [#51718](https://github.com/aarnphm/aarnphm.github.io/issues/51718) and add a small shared-storage BLHNC test that covers the heterogeneous-TP case? That would protect the exact failure we’ve been seeing in CI.

**LucasWilkinson** (2026-08-21T15:35:15Z):

> We’ve been seeing a strange CrossLayer KV accuracy failure in AMD CI. In \[[build 12283](https://buildkite.com/vllm/amd-ci/builds/12283#01a0208f-90ca-4679-ad77-da1ebadb049c)\](<https://buildkite.com/vllm/amd-ci/builds/12283#01a0208f-90ca-4679-ad77-da1ebadb049c>), Qwen worked normally when the prefiller and decoder were both TP2, but TP1 prefiller <span>&rarr;</span> TP2 decoder completed every request successfully and still got exactly `0.0` accuracy.
>
> It initially looked like an intermittent runner or dependency problem, but the same TP-specific failure reproduced elsewhere. My understanding is that the standardized layout caused regular BLHNC cache storage to take the packed-MLA registration path, which gives NIXL the wrong offsets when the P/D TP sizes differ. The change in `9b253bc6` keeps that packed path limited to MLA and appears to fix the actual corruption.
>
> After that change, the same job passed all six Qwen and DeepSeek configurations in \[[build 12288](https://buildkite.com/vllm/amd-ci/builds/12288#01a02108-4e24-46f5-9595-d8b18a85b24e)\](<https://buildkite.com/vllm/amd-ci/builds/12288#01a02108-4e24-46f5-9595-d8b18a85b24e>). Could we keep this fix in [#51718](https://github.com/aarnphm/aarnphm.github.io/issues/51718) and add a small shared-storage BLHNC test that covers the heterogeneous-TP case? That would protect the exact failure we’ve been seeing in CI.

I believe we hit this in [AMD build 12283](https://buildkite.com/vllm/amd-ci/builds/12283#01a0208f-90ca-4679-ad77-da1ebadb049c), and it is now passing in [AMD build 12288](https://buildkite.com/vllm/amd-ci/builds/12288#01a02108-4e24-46f5-9595-d8b18a85b24e) after [`9b253bc6`](https://github.com/vllm-project/vllm/pull/51718/changes/9b253bc6e25752b32cb724b331cfa58f1a882dcb) preserved the fix by restricting the packed-storage path to MLA.

**LucasWilkinson** (2026-08-21T15:46:48Z):

/ci retry

**github-actions** (2026-08-21T15:47:04Z):

✅ The previous CI build is still running: <https://buildkite.com/vllm/ci/builds/84920>

**mergify** (2026-08-21T15:52:28Z):

Hi @LucasWilkinson, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**njhill** (2026-08-21T16:26:34Z):

/ci retry

**github-actions** (2026-08-21T16:26:52Z):

✅ The previous CI build is still running: <https://buildkite.com/vllm/ci/builds/84920>

**njhill** (2026-08-21T17:19:22Z):

/ci retry

**github-actions** (2026-08-21T17:19:38Z):

✅ The previous CI build is still running: <https://buildkite.com/vllm/ci/builds/84920>

**LucasWilkinson** (2026-08-21T17:25:57Z):

/ci retry

**github-actions** (2026-08-21T17:26:14Z):

✅ The previous CI build is still running: <https://buildkite.com/vllm/ci/builds/84920>

**LucasWilkinson** (2026-08-21T18:28:58Z):

/ci run all

**github-actions** (2026-08-21T18:29:17Z):

✅ Triggered [Buildkite CI #85074](https://buildkite.com/vllm/ci/builds/85074) for commit `e16091ff05a7`.

**LucasWilkinson** (2026-08-21T20:28:39Z):

/ci run all

**github-actions** (2026-08-21T20:28:57Z):

✅ Triggered [Buildkite CI #85105](https://buildkite.com/vllm/ci/builds/85105) for commit `1cd6bbdf70d5`.

**LucasWilkinson** (2026-08-21T20:35:33Z):

/ci run all

**github-actions** (2026-08-21T20:35:51Z):

✅ Triggered [Buildkite CI #85107](https://buildkite.com/vllm/ci/builds/85107) for commit `92271c89cacd`.

**LucasWilkinson** (2026-08-21T20:42:45Z):

/ci run all

**github-actions** (2026-08-21T20:43:06Z):

✅ Triggered [Buildkite CI #85110](https://buildkite.com/vllm/ci/builds/85110) for commit `9acfea86f1e9`.

**LucasWilkinson** (2026-08-21T20:44:47Z):

/amd-ci run nightly

**github-actions** (2026-08-21T20:45:05Z):

✅ Triggered [Buildkite AMD CI #12317](https://buildkite.com/vllm/amd-ci/builds/12317) for commit `9acfea86f1e9`.

**AndreasKaratzas** (2026-08-21T21:36:00Z):

weird segfault here:
<https://buildkite.com/vllm/ci/builds/85110/list?jid=01a02611-7117-4292-8872-099714e18175&tab=output#L1665>

investigating

**njhill** (2026-08-21T22:07:38Z):

@AndreasKaratzas looks like the retry passed…

**AndreasKaratzas** (2026-08-21T23:22:50Z):

@njhill yeah i m investigating this in parallel and looks like a very weird case. Wouldn’t attribute this intermittent error to this PR as well. Also the AMD ci looks greener than the nightly so i m confident about this RP 😅 I am doing some final checks and will stamp this too.

**njhill** (2026-08-22T00:11:55Z):

I have confirmed that the remaining failures are unrelated, also fail identically on other builds.

**njhill** (2026-08-22T00:15:33Z):

omg there is a new conflict

**njhill** (2026-08-22T00:33:36Z):

For reference here is the good CI run (run all) <https://buildkite.com/vllm/ci/builds/85110> - which had only 4 confirmed-unrelated failures.

The only difference is that I rebased and included a revert of [vllm-project/vllm#53002](https://github.com/vllm-project/vllm/pull/53002) which was merged after the above CI run.

**njhill** (2026-08-22T00:37:02Z):

/ci run

**github-actions** (2026-08-22T00:37:20Z):

✅ Triggered [Buildkite CI #85135](https://buildkite.com/vllm/ci/builds/85135) for commit `94c9d6c9d748`.

**mgoin** (2026-08-22T01:13:21Z):

Let’s goooooo!

