---
title: '51704'
pageLayout: default
slug: thoughts/inference/vllm-prs/51704
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/51704.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#51704](https://github.com/aarnphm/aarnphm.github.io/issues/51704) \[5/N\]\[KV-Cache Layout Refactor\] Backend-published KV packing via customize_spec

[vllm-project/vllm#51704](https://github.com/vllm-project/vllm/pull/51704) · LucasWilkinson · MERGED · created 2026-08-10T16:20:40Z · merged 2026-08-14T13:47:56Z

## Purpose

Part of the KV-cache layout standardization series (RFC [#42082](https://github.com/aarnphm/aarnphm.github.io/issues/42082)). **Stacked on [#51612](https://github.com/aarnphm/aarnphm.github.io/issues/51612)** — the diff shown includes it until that lands and this retargets `main`.

Attention specs today carry quant-format sizing knowledge inline: `nvfp4` / per-token-head branches in the page-size properties, a `TQFullAttentionSpec` subclass, and fp8_ds_mla constants in MLA spec overrides. This PR makes specs plain data and moves each packed format to the backend/component that owns it:

- `AttentionSpec` gains two optional packing fields `num_head_slots` (H in standard layout) and`state_content_bytes` (C in standard layout). Page sizes derive uniformly as `num_heads × storage_block_size × content`.
  > NOTE: i dont love the num_head_slots name but this should hopefully be temporary until we can [fully refactor the backends](https://github.com/vllm-project/vllm/issues/42449); i.e. separate allocation from connector and kv-cache manager concerns
- `AttentionBackend.customize_spec(spec)` is a temporary hook until we can move to having the backend return the spec instead of the layer (see: [vllm-project/vllm#42449](https://github.com/vllm-project/vllm/issues/42449))

Prepares the ground for the layout-standardization PR ([[6/N]]\([vllm-project/vllm#51718](https://github.com/vllm-project/vllm/pull/51718))), where these fields describe the standardized `[B, H, N, C]` page and the AITER backends publish separate K/V head groups through the same hook.

---

AI-assisted (Claude); reviewed by submitter

---

## comments

**mergify** (2026-08-11T19:36:28Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**LucasWilkinson** (2026-08-11T22:08:51Z):

/ci run

**github-actions** (2026-08-11T22:09:09Z):

✅ Triggered [Buildkite CI #83451](https://buildkite.com/vllm/ci/builds/83451) for commit `5a24c75ba765`.

**njhill** (2026-08-12T00:25:27Z):

@LucasWilkinson this CI failure looks like it could be related:



```
FAILED v1/worker/test_dsv4_packed_zeroer_geometry.py::test_dsv4_packed_zeroer_geometry - RuntimeError: shape '[100, 64, 584]' is invalid for input of size 3283200
```

**LucasWilkinson** (2026-08-12T03:47:55Z):

/ci run

**github-actions** (2026-08-12T03:48:55Z):

✅ Triggered [Buildkite CI #83498](https://buildkite.com/vllm/ci/builds/83498) for commit `56778231b451`.

**mergify** (2026-08-12T16:12:36Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**LucasWilkinson** (2026-08-12T16:27:29Z):

/ci run

**github-actions** (2026-08-12T16:27:46Z):

✅ Triggered [Buildkite CI #83577](https://buildkite.com/vllm/ci/builds/83577) for commit `d90e93ff17f7`.

**mergify** (2026-08-12T16:34:52Z):

Hi @LucasWilkinson, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**AndreasKaratzas** (2026-08-12T17:55:54Z):

AMD is not triggering all tests for some reason here. Let me trigger a manual build and if it is possible, let’s wait for it to finish before merging.

**LucasWilkinson** (2026-08-12T18:07:39Z):

/ci run

**github-actions** (2026-08-12T18:08:40Z):

✅ Triggered [Buildkite CI #83593](https://buildkite.com/vllm/ci/builds/83593) for commit `ab2a4342a19b`.

**AndreasKaratzas** (2026-08-12T18:27:37Z):

Here is the full build btw:

- <https://buildkite.com/vllm/amd-ci/builds/11976/canvas>

**LucasWilkinson** (2026-08-12T21:37:17Z):

/ci run

**github-actions** (2026-08-12T21:37:34Z):

✅ Triggered [Buildkite CI #83613](https://buildkite.com/vllm/ci/builds/83613) for commit `aedea6d2a520`.

**LucasWilkinson** (2026-08-13T06:15:07Z):

/ci run

**github-actions** (2026-08-13T06:15:26Z):

✅ Triggered [Buildkite CI #83682](https://buildkite.com/vllm/ci/builds/83682) for commit `ca462022a34f`.

**LucasWilkinson** (2026-08-13T13:44:52Z):

/ci retry

**github-actions** (2026-08-13T13:45:16Z):

✅ No failed, timed-out, or expired jobs need retrying: <https://buildkite.com/vllm/ci/builds/83682>

**njhill** (2026-08-13T15:19:37Z):

The arm-cpu failure is a known one which has already been fixed on main.

**LucasWilkinson** (2026-08-13T16:41:42Z):

[buildkite/ci/pr/arm-cpu-test](https://buildkite.com/vllm/ci/builds/83682#019ffb5e-3ec8-4e14-a3e7-149092bf6eea) fixed on main [vllm-project/vllm#52092](https://github.com/vllm-project/vllm/pull/52092)
[buildkite/ci/pr/gpqa-eval-gpt-oss-dgx-spark](https://buildkite.com/vllm/ci/builds/83682#019ffbb5-df0e-40f2-a4be-52e1310afe08) also failing on main, fix here: [vllm-project/vllm#52148](https://github.com/vllm-project/vllm/pull/52148)

**AndreasKaratzas** (2026-08-13T16:55:55Z):

These are new regressions from this PR:

- <https://buildkite.com/vllm/amd-ci/builds/11990/list?sid=019ff9c2-c495-4c60-be8a-63be93a4ba34&tab=output>
- <https://buildkite.com/vllm/amd-ci/builds/11990/list?sid=019ff9c2-c496-43d9-b509-f0bcfe62d759&tab=output>
- <https://buildkite.com/vllm/amd-ci/builds/11990/list?jid=019ff9c2-c57b-4e1d-9fa0-2a359eeb3125&tab=output>

These failures are not on our nightly:

- <https://buildkite.com/vllm/amd-ci/builds/11990/list?sid=019ff9c2-c439-47ba-89ed-c70d5846d208&tab=output>
- <https://buildkite.com/vllm/amd-ci/builds/11990/list?sid=019ff9c2-c4a3-43f4-94d9-4bc31ccebcdd&tab=output>

Before we merge this PR, can we ensure that there is parity with AMD nightly CI?

UPDATE: Actually in the very latest nightly the Kernel test groups indeed fail. There is a PR to fix them here: [vllm-project/vllm#52139](https://github.com/vllm-project/vllm/pull/52139)

**LucasWilkinson** (2026-08-13T18:10:52Z):

/ci run

**github-actions** (2026-08-13T18:11:41Z):

✅ Triggered [Buildkite CI #83775](https://buildkite.com/vllm/ci/builds/83775) for commit `6aae82473a1a`.

**mergify** (2026-08-14T02:13:46Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**LucasWilkinson** (2026-08-14T02:52:51Z):

/ci run

**github-actions** (2026-08-14T02:53:07Z):

✅ Triggered [Buildkite CI #83828](https://buildkite.com/vllm/ci/builds/83828) for commit `4f72e918fbb5`.

**AndreasKaratzas** (2026-08-14T04:28:21Z):

From AMD tests perspective, everything looks good :)

