---
title: '56629'
pageLayout: default
slug: thoughts/inference/vllm-prs/56629
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56629.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56629](https://github.com/aarnphm/aarnphm.github.io/issues/56629) \[5/N\] Share HiSparse host cache across TP ranks (reopens [#52760](https://github.com/aarnphm/aarnphm.github.io/issues/52760))

[vllm-project/vllm#56629](https://github.com/vllm-project/vllm/pull/56629) · LucasWilkinson · MERGED · created 2026-09-12T18:20:27Z · merged 2026-09-12T23:48:25Z

## Summary

Share the replicated HiSparse MLA host cache across local TP ranks using one mmap-backed pool. TP rank 0 writes the shared pool; peers wait on its IPC events before reading it. Other executor and parallel layouts retain private pools.

This reopens [#52760](https://github.com/aarnphm/aarnphm.github.io/issues/52760) against `main`. The original PR was automatically closed when \[3/N\] ([#53781](https://github.com/aarnphm/aarnphm.github.io/issues/53781)) merged and its base branch, `feat/hisparse-mla-decode-main`, was deleted.

I am not the original author of this work; I reopened the existing PR so it can continue through review. Credit goes to **Chao Lei (@LCAIZJ)**, the original PR author, and **Matthew Bonanni (@MatthewBonanni)** for their implementation and follow-up work. The original commit authorship, co-author credits, and sign-offs are preserved, including my previous follow-up fixes.

The 11 original shared-host-cache commits rebased cleanly onto `main` with unchanged patches. The preceding metrics commit is already covered by merged [#56061](https://github.com/aarnphm/aarnphm.github.io/issues/56061). A follow-up simplifies shared-region initialization as described below.

## Implementation

- Resolve shared/private host-pool mode and the physical host-block stride in the cache planner and propagate them through `KVCacheConfig`.
- Check physical host RAM in the mmap creator only and populate the shared backing once.
- Reuse `SharedOffloadRegion`’s existing barrier with the TP CPU group instead of introducing file locks. The creator populates before the barrier; the file is unlinked after all workers have mapped it, protecting against backing-file leaks after SIGKILL.
- Keep the ordinary offload population path close to `main`. The shared-region diff is reduced from 164 changed lines to 85.
- Preserve initialization rollback and cleanup of every successful CUDA host registration. Registration boundaries remain aligned with transfer geometry.
- Extend the existing multiprocess test to cover replicated-slot visibility, creator-only population, and cleanup after SIGKILL; remove the redundant standalone sharing test.
- Correct the documentation to describe the single TP writer and remove the stale claim that `num_gpu_blocks_override` overrides host capacity.

## Duplicate-work check

This replaces closed [#52760](https://github.com/aarnphm/aarnphm.github.io/issues/52760), which itself replaced closed [#51324](https://github.com/aarnphm/aarnphm.github.io/issues/51324). Checked both PR discussions, open PRs referencing either number, and open HiSparse/shared-host-cache work before opening this replacement; no open PR carrying the same change was found.

## Validation

- `CUDA_VISIBLE_DEVICES='' .venv/bin/python -m pytest tests/v1/kv_offload/cpu/test_shared_offload_region.py -q` — 46 passed on the current implementation.
- `CUDA_VISIBLE_DEVICES='' .venv/bin/python -m pytest tests/v1/worker/test_kv_cache_allocation_scope.py -q` — 4 passed. Fixed the outdated zero-argument `HiSparseHostPool` mock to accept the two config arguments introduced by this PR.
- `chg run -g 1 --timeout 15m -- .venv/bin/python -m pytest tests/v1/worker/test_utils.py tests/v1/worker/test_attn_utils.py tests/v1/worker/test_kv_cache_allocation_scope.py tests/v1/core/test_kv_cache_utils.py tests/v1/kv_connector/unit/offloading_connector/test_config.py -q` — 241 passed; two PP configuration tests failed because they require more than one visible GPU.
- `chg run -g 4 --wait 2m --timeout 5m -- .venv/bin/python -m pytest tests/v1/core/test_kv_cache_utils.py::test_get_kv_cache_config_mamba_hybrid_sharing_pp_group_count_bump tests/v1/core/test_kv_cache_utils.py::test_get_kv_cache_config_mamba_hybrid_sharing_pp_starved_stage -q` — both passed. All 243 tests in the related suites therefore passed with the required GPU visibility.
- Local TP=2 CUDA smoke test through real vLLM distributed groups and `allocate_hisparse_host_pools`: all four cases passed (single/split registration, each across two allocation/cleanup cycles). Verified shared inode, early unlink, rank-0 GPU-to-host writes, IPC-event waits, exact host-to-GPU round trips on both ranks, and registration cleanup. Used a 256 KiB backing and a scaled 40 KiB registration limit to exercise seams; this does not test production-sized registrations.
- Pre-commit, including Ruff and mypy, passed for the changed files. `git diff --check` passed.
- No test processes or GPU allocations remained on the reserved GPUs after completion.

No end-to-end TP model evaluation or large-pool performance measurement was run; these tests do not establish model-output parity or startup performance at production pool sizes.

AI assistance (OpenAI Codex) was used for the rebase, review, follow-up changes, tests, and PR description. Existing AI co-author credits remain in the commits.

---

## comments

**mergify** (2026-09-12T18:21:01Z):

Documentation preview: [https://vllm—56629.org.readthedocs.build/en/56629/](https://vllm--56629.org.readthedocs.build/en/56629/)

**LucasWilkinson** (2026-09-12T19:47:25Z):

/ci run

**github-actions** (2026-09-12T19:47:42Z):

✅ Triggered [Buildkite CI #88557](https://buildkite.com/vllm/ci/builds/88557) for commit `c2d9f3ea2c87`.

**LucasWilkinson** (2026-09-12T20:23:54Z):

/ci run

**github-actions** (2026-09-12T20:24:13Z):

✅ Triggered [Buildkite CI #88560](https://buildkite.com/vllm/ci/builds/88560) for commit `1acc57e8450d`.

