---
title: '49226'
pageLayout: default
slug: thoughts/inference/vllm-prs/49226
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/49226.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#49226](https://github.com/aarnphm/aarnphm.github.io/issues/49226) \[Bugfix\]\[KVConnector\] Disable cross-layer KV blocks for per-token-head quant

[vllm-project/vllm#49226](https://github.com/vllm-project/vllm/pull/49226) · Achyuthan-S · MERGED · created 2026-07-20T16:42:10Z · merged 2026-07-26T02:24:28Z

## Purpose

Fixes [#48412](https://github.com/aarnphm/aarnphm.github.io/issues/48412).

Combining `OffloadingConnector` with a per-token-head quantized KV cache
(`fp8_per_token_head` / `int8_per_token_head` / `int4_per_token_head`) corrupts
generations on the **first forward pass**, before any offloaded block is ever
restored.

### Root cause

`OffloadingConnector.prefer_cross_layer_blocks` is hardcoded `True`. For
single-group models on backends with `indexes_kv_by_block_stride` (e.g.
`TRITON_ATTN`), the model runner therefore allocates a **cross-layer** KV cache
(`allocate_uniform_kv_caches` in `kv_connector_model_runner_mixin.py`): one
contiguous buffer shared by all layers.

Per-token-head quant stores an inline fp32 scale in a padded tail of each
`(head, token)` cell. The attention backend carves strided fp32 views over
those pad bytes in `_ensure_scale_caches` (`triton_attn.py`), assuming a
**per-layer, contiguous** buffer. Against the interleaved cross-layer
allocation, the assumed block stride is wrong and every layer’s scale views
collide at the same offset, so writing scales aliases neighboring K/V data and
corrupts the cache from the first decode.

### Fix

Return `prefer_cross_layer_blocks = False` when the KV cache dtype uses
per-token-head scales, so these models take the **per-layer** offload path
(which lays out each cell’s scale pad correctly). Non-per-token-head configs —
plain `fp8` included — are unchanged and still prefer the cross-layer layout.

This is a soft fallback, not a hard failure: offloading still works for these
dtypes via the per-layer path — only the cross-layer transfer optimization is
skipped, and only for per-token-head dtypes. Teaching the cross-layer allocation
and `_ensure_scale_caches` to pack scales in the shared buffer would restore that
optimization, but is a larger, hotter-path change and can be a follow-up.

### Relationship to [#48411](https://github.com/aarnphm/aarnphm.github.io/issues/48411) (merged)

[#48411](https://github.com/aarnphm/aarnphm.github.io/issues/48411) fixed the **per-layer** offload _transfer width_ — inline scales were
truncated mid-cell on restore. It explicitly deferred the **cross-layer**
allocation incompatibility to [#48412](https://github.com/aarnphm/aarnphm.github.io/issues/48412), which this PR addresses. With [#48411](https://github.com/aarnphm/aarnphm.github.io/issues/48411)
already in `main`, this PR completes per-token-head + offloading support:
models fall back to the (now-correct) per-layer path instead of the corrupting
cross-layer allocation. No other open PR addresses [#48412](https://github.com/aarnphm/aarnphm.github.io/issues/48412).

## Test Plan



```bash
pytest tests/v1/kv_connector/unit/offloading_connector/test_cross_layer_preference.py -v
```

cc @Etelis @orozery

---

## comments

**Achyuthan-S** (2026-07-21T04:35:55Z):

Pushed an update addressing both comments: the per-token-head check is now a single gate in use_uniform_kv_cache() (keyed on kv_quant_mode.is_per_token_head) instead of a per-connector override, so it covers every connector that prefers cross-layer blocks. offloading_connector.py is reverted to upstream; net change is +4 lines in the mixin plus a test in test_kv_cache_layout.py. Also rebased onto latest main.

**Etelis** (2026-07-21T06:14:16Z):

LGTM

@orozery

**orozery** (2026-07-21T16:23:51Z):

Thanks @Achyuthan-S ! Looks good to me.
I don’t think we need the tests. Can we remove them?

**Achyuthan-S** (2026-07-21T17:25:25Z):

Hey @orozery , removed — the PR is now just the gate in use_uniform_kv_cache. End-to-end behavior was verified on H100 in @Etelis’s review above.

**mergify** (2026-07-21T17:29:21Z):

Hi @Achyuthan-S, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**Achyuthan-S** (2026-07-22T05:43:41Z):

The 6 failures are infra (exit status -1 = agent-lost/cancellation; two logs show “Received cancellation signal”, pytorch-compilation was mid-pass). No traceback references this change, and the KV-connector lanes (v1-core-plus-kv, v1-others-cpu, nixlconnector-2gpu) all passed.

Could a maintainer re-trigger the failed jobs?

**Achyuthan-S** (2026-07-22T09:51:17Z):

The remaining v1-sample-plus-logits failure is unrelated: the 5 failures are test_logprobs_mode with gpu_memory_utilization=0.05 and 2 other LLMs alive on a 16 GB MIG slice — the engine core proc dies at startup under memory pressure (Failed core proc(s): \{\}), free mem dropping 15.7→6.8 GiB across the stacked LLMs. No traceback references this change, it uses no KV connector, and the 294 other logprobs tests + all KV lanes passed. Should be clear on a re-run of that job.

**Achyuthan-S** (2026-07-25T07:26:35Z):

Small update .

The 24 failures are not from this change (still +4 lines, 1 file; no failing log references it):

- 20/24 are Buildkite infra — the x86 agent-environment hook failed at setup (Cannot perform an interactive login from a non TTY device), so tests never ran.
- 4/24 are pre-existing main regressions in unrelated areas (test_model_arch_config\[Zamba2\] dtype assert, test_compile_correctness, granite-4.0 test_hybrid, test_prompt_logprobs_e2e shape error) — none touch use_uniform_kv_cache.

