---
title: '47314'
pageLayout: default
slug: thoughts/inference/vllm-prs/47314
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/47314.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#47314](https://github.com/aarnphm/aarnphm.github.io/issues/47314) \[BugFix\] Fix packed HND KV cache reshape for FlashAttention

[vllm-project/vllm#47314](https://github.com/vllm-project/vllm/pull/47314) · LucasWilkinson · MERGED · created 2026-07-01T17:09:33Z · merged 2026-07-11T03:22:39Z

Fixes [#47054](https://github.com/aarnphm/aarnphm.github.io/issues/47054).

Summary:

- Preserve packed KV cache physical layout before permuting back to backend logical shape.
- This keeps FlashAttention HND caches logically shaped as `[blocks, 2, block_size, heads, dim]`.

Repro used during debugging:



```bash
MODEL="openai/gpt-oss-120b"
TP_SIZE=2
CPU_BYTES=26843545600
KV_TRANSFER_CONFIG=$(cat <<EOF_JSON
{
  "kv_connector": "OffloadingConnector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {
    "enable_cross_layers_blocks": "True",
    "spec_name": "CPUOffloadingSpec",
    "cpu_bytes_to_use": ${CPU_BYTES},
    "eviction_policy": "lru"
  }
}
EOF_JSON
)
PATH="$PWD/.venv/bin:$PATH" CUDA_VISIBLE_DEVICES=1,2 .venv/bin/vllm serve "${MODEL}" \
  --host 127.0.0.1 \
  --port 8010 \
  --tensor-parallel-size="${TP_SIZE}" \
  --kv-transfer-config "${KV_TRANSFER_CONFIG}" \
  --enable-prefix-caching \
  --no-disable-hybrid-kv-cache-manager \
  --num-gpu-blocks-override 16384
```



```bash
.venv/bin/lm_eval run \
  --model local-completions \
  --model_args model=openai/gpt-oss-120b,base_url=http://127.0.0.1:8010/v1/completions,tokenized_requests=False,num_concurrent=8,max_retries=1,timeout=600,tokenizer_backend=huggingface \
  --tasks gsm8k \
  --limit 100 \
  --num_fewshot 5 \
  --gen_kwargs temperature=0,max_gen_toks=256
```

Tests:

- tested above repro
- `.venv/bin/python -m pytest tests/v1/worker/test_attn_utils.py -q`
- `pre-commit run ruff-check --files vllm/v1/worker/gpu/attn_utils.py tests/v1/worker/test_attn_utils.py`
- commit hooks

AI assistance was used to prepare this change. Reviewed by the submitter.

---

## comments

