---
title: '53781'
pageLayout: default
slug: thoughts/inference/vllm-prs/53781
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/53781.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#53781](https://github.com/aarnphm/aarnphm.github.io/issues/53781) \[3/N\] HiSparse: host-resident sparse-MLA decode hot-buffering

[vllm-project/vllm#53781](https://github.com/vllm-project/vllm/pull/53781) · MatthewBonanni · MERGED · created 2026-08-25T19:21:03Z · merged 2026-09-12T04:36:47Z

## Summary

HiSparse adds a host-resident tier beneath vLLM’s normal GPU KV cache for sparse-MLA decode. Unlike the original always-host-resident design in [#46326](https://github.com/aarnphm/aarnphm.github.io/issues/46326), this implementation keeps as much KV as possible in the normal device cache and spills pages to pinned host memory only when GPU capacity must be reclaimed.

Supersedes [#46326](https://github.com/aarnphm/aarnphm.github.io/issues/46326).

## Design overview



```text
resident GPU pages
        |
        | spill under pressure
        v
  pinned host pool ---- selected top-k misses ----> per-request GPU hot buffer
        ^                                                     |
        |                                                     v
        +---------------- prefix reuse ---------------- sparse attention
```

The active indexer cache remains device-resident and produces the logical top-k positions needed by sparse attention. A fused CUDA resolver maps each selected position through three possible locations:

1. a normal resident GPU page;
2. an existing row in the request’s GPU hot buffer;
3. the pinned host pool.

Resident hits return before the hot-cache lookup. Hot hits update a GPU-side LRU, while misses replace LRU entries by gathering only the selected host rows. Resolution, replacement, and LRU updates remain on the GPU and are compatible with CUDA graph replay.

HiSparse does not maintain a private host-side copy of indexer KV. The indexer remains a normal prefix-cacheable GPU cache group. When OffloadingConnector is configured alongside HiSparse, generic KV offloading preserves and restores the indexer while HiSparse continues to own only sparse-MLA residency. If the sparse host prefix extends beyond the available indexer prefix, the scheduler restores only the missing indexer suffix, capped at the sparse host boundary. Without generic offloading, the request falls back to the shorter shared prefix and recomputes the remainder.

## Residency and ownership

The normal KV cache manager continues to own GPU block allocation. `HiSparseCoordinator` owns logical host blocks, host-prefix identity, and residency transitions.

When GPU pressure requires reclaiming a resident page, the coordinator reserves its host destination and sends a transfer command to the worker. The resident GPU block becomes reusable after every worker has acknowledged that the copy is enqueued in stream order. The host page is not published for prefix reuse until every copy has completed.

The main components are:

- `HiSparseCoordinator`: scheduler-side host allocation, prefix state, and spill policy.
- `HiSparseConnector`: carries residency commands and transfer acknowledgements between scheduler and workers.
- `HiSparseConnectorWorker`: manages the pinned host-pool lifecycle and executes copies.
- `HiSparseRuntime`: owns each cache’s host/hot views, GPU LRU state, and fused attention-time resolution.

Host blocks use the same token hashes as their resident counterparts, but their hashes are published only after the contents are durable. Spilling therefore preserves prefix-cache value without allowing an incomplete host write to be reused.

## P/D and NIXL integration

NIXL remains the P/D transport; HiSparse determines the decoder-side landing policy. The decoder chooses one policy per request during admission:

- If the imported prefix fits in the normal GPU pools, NIXL writes it into resident device pages.
- Otherwise, the bulk sparse-MLA KV lands in the registered HiSparse host pool. Device-only regions such as the indexer are still imported to GPU, the request receives its fixed hot-buffer allocation, and pages needed immediately are mirrored into resident GPU pages.

The two generic connector prerequisites below are intentionally separate from HiSparse:

- [#53779](https://github.com/aarnphm/aarnphm.github.io/issues/53779) lets connectors select externally transferable cache groups while retaining their scheduler group IDs.
- [#53780](https://github.com/aarnphm/aarnphm.github.io/issues/53780) carries stride, capacity, block size, group mapping, and identity per NIXL region, allowing P/D peers with different layouts and memory placement.

[#53263](https://github.com/aarnphm/aarnphm.github.io/issues/53263) is a separate, optional optimization for same-node P/D. It stages device-to-host NIXL reads through decoder GPU memory to avoid UCX’s slow TCP loopback path. It is not required by the intended multi-node P/D design.

## Scope

This PR implements the NVIDIA sparse-MLA path and includes the staging required for correct local or mixed prefill on a decode worker. Metrics are split into [#53782](https://github.com/aarnphm/aarnphm.github.io/issues/53782), and DeepSeek V4’s compressed cache geometry is split into [#53592](https://github.com/aarnphm/aarnphm.github.io/issues/53592).

## Preliminary results

Preliminary B300 results:

<img width="1558" height="939" alt="glm_20k_10k_20260820_current" src="https://github.com/user-attachments/assets/cc7a519c-0a04-4aff-a24a-cb3b92c4d800" />
<img width="1558" height="939" alt="glm_32k_8k_20260820_current" src="https://github.com/user-attachments/assets/2c1bbacd-0011-40fe-b31f-781d4e90e681" />
<img width="1558" height="939" alt="glm_60k_10k_20260820_current" src="https://github.com/user-attachments/assets/01202f88-85d5-4fd7-baf1-4da2bd5cfeaa" />

AI assistance was used in the preparation of this PR.

---

## comments

**mergify** (2026-08-25T19:21:42Z):

Documentation preview: [https://vllm—53781.org.readthedocs.build/en/53781/](https://vllm--53781.org.readthedocs.build/en/53781/)

**mergify** (2026-08-25T19:33:58Z):

Hi @MatthewBonanni, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**MatthewBonanni** (2026-08-25T19:59:47Z):

/ci run

**github-actions** (2026-08-25T20:00:06Z):

✅ Triggered [Buildkite CI #85557](https://buildkite.com/vllm/ci/builds/85557) for commit `f1542536ad1c`.

**mergify** (2026-08-26T14:08:14Z):

Hi @MatthewBonanni, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-08-26T14:34:56Z):

Hi @MatthewBonanni, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**samsja** (2026-08-28T18:31:08Z):

Ran this stack (up through 5/N, `feat/nixl-host-read-staging` @ [`dfc3f67`](https://github.com/aarnphm/aarnphm.github.io/commit/dfc3f675e)) on GLM-5.2-FP8 / GLM-5.3 (`glm_moe_dsa`, 78L, index_topk=2048) in a disaggregated P/D deployment on 8xH200 nodes — 2 prefill + 2 decode engines (tp=8, dp=1 each), NIXL over source-built UCX 1.19, nixl 0.10.1, torch 2.13.0+cu130, `hisparse_config.host_pool_gib` set on both roles. Single-node HiSparse works great; P/D needed two workarounds and has one open correctness issue. Reporting all three since GLM-family shapes don’t seem covered by CI:

**1. NIXL region geometry breaks on GLM’s packed cache pool (startup crash).**
GLM-5.2/5.3 allocate the indexer/resident/hot caches in one shared storage. In `NixlBaseConnectorWorker.register_kv_caches`, the packed-storage MLA branch registers whole-row regions (`storage_block_len = registration_len // num_blocks`), so a `hisparse_resident` layer’s region is the full packed row — 4x its page here — and `HiSparseNixlAdapter.register_region` rejects it:



```
ValueError: HiSparse host page is smaller than its transfer region:
layer=model.layers.0.self_attn.attn.hisparse_resident, host_stride=73728, region_block_len=294912
```

`block_size` doesn’t change it (tried 64, the e2e-test value). Local workaround: for `hisparse_resident` layers, emit per-cache regions `(cache.data_ptr(), physical_page_size, row_stride)` instead. This may not be the right general fix (see finding 3).

**2. Host-pool NIXL registration includes unpinned slop → UCX “already mapped” (startup crash).**
`allocate_pinned_host_pool` over-allocates `padded_size + page` and `cudaHostRegister`s only the page-aligned slice, but `register_region` records the _whole_ backing storage in `_host_registration_ranges`. UCX’s cuda_copy MD then `cuMemHostRegister`s a range that overlaps the already-pinned middle and fails hard:



```
UCX ERROR cuMemHostRegister_v2(...) failed: part or all of the requested memory range is already mapped
nixl_cu13._bindings.nixlBackendError: NIXL_ERR_BACKEND
```

Reproduces at any `host_pool_gib` (tried 4 and 24). Local workaround: shrink each recorded range to the pinned sub-range (`base + (-base % 4096)`, `nbytes - 4096`) before `register_memory` — UCX’s memory-type probe then dedupes. (Skipping the pool’s own pinning instead is rejected by the `hisparse_backup` kernel’s “host_cache must be pinned” check.)

**3. Open correctness issue: long-prefix P/D transfers garble output on GLM shapes.**
With the two workarounds above, the P/D fleet serves and _short_ routed requests are byte-correct, but long-prefix requests (e.g. 40k-token prompt) return garbled text through the router while the same prompt direct-to-decode (local prefill) is coherent — i.e. multi-block transfer of hisparse KV corrupts data. This may well be my workaround for finding 1 mis-addressing blocks under `_physical_blocks_per_logical_kv_block` packing rather than a bug in the stack itself — but either way the GLM packed-pool P/D path appears unexercised upstream. Happy to test a proper fix or provide more detail (logs, configs, exact geometry dumps).

Also worth a docs note: the NIXL handshake (`Number of KV layers must match between prefill and decode`) means `hisparse_config` must be enabled on **both** P/D roles, not just the decoder — enabling decode-only fails at the first transfer.

Great work on this stack overall — the P/D win on our setup was 40k-token prefill in 10s via prefill nodes vs 3m20s decode-local, and the HMA path works exactly as documented on a single node.

**coderabbitai** (2026-09-03T21:22:45Z):

<!-- This is an auto-generated comment: summarize by coderabbit.ai -->

<!-- review_stack_entry_start -->

[![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/vllm-project/vllm/pull/53781)

<!-- review_stack_entry_end -->

<!-- recent_review_start -->

No actionable comments were generated in the recent review. 🎉

<details>
<summary>ℹ️ Recent review info</summary>

<details>
<summary>⚙️ Run configuration</summary>

**Configuration used**: Repository UI

**Review profile**: CHILL

**Plan**: Team

**Run ID**: `6ec071f0-bca0-44ef-a245-743135f8c2b8`

</details>

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between [`a6ccaa1`](https://github.com/aarnphm/aarnphm.github.io/commit/a6ccaa1984cfd92eaac77a44f20316198c0e582b) and [`49d1fae`](https://github.com/aarnphm/aarnphm.github.io/commit/49d1fae38f6f1d6175f8abf49c74f1e0040a92c6).

</details>

<details>
<summary>📒 Files selected for processing (5)</summary>

- `tests/test_config.py`
- `tests/v1/attention/test_sparse_mla_backends.py`
- `tests/v1/core/test_prefix_caching.py`
- `tests/v1/kv_connector/unit/offloading_connector/test_config.py`
- `tests/v1/worker/test_utils.py`

</details>

<details>
<summary>💤 Files with no reviewable changes (1)</summary>

- tests/v1/kv_connector/unit/offloading_connector/test_config.py

</details>

**Included review availability:** Your plan provides up to 10 included reviews per hour; 9 remain after this review.

</details>

---

<!-- recent_review_end -->

<!-- walkthrough_start -->

## Walkthrough

HiSparse adds host-backed sparse MLA execution, CUDA residency kernels, multi-pool KV-cache allocation, scheduler and connector coordination, model wiring, warmup support, and broad unit and end-to-end coverage. It also updates transfer APIs and FP8 cache gathering.

### Changes

**HiSparse host-tier cache**

| Layer / File(s)                                                                                                                                                                                | Summary                                                                                                                                                                                                               |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Configuration, cache contracts, and allocation** <br> `vllm/config/*`, `vllm/v1/kv_cache_interface.py`, `vllm/v1/core/*`                                                                     | Adds HiSparse configuration, cache roles, host and resident cache specifications, multiple block pools, host-capacity validation, residency tracking, reclamation, prefix materialization, and scheduler integration. |
| **CUDA kernels and runtime** <br> `csrc/libtorch_stable/*`, `vllm/v1/hisparse/*`, `vllm/v1/attention/backends/mla/*`                                                                           | Adds CUDA residency, invalidation, gather, and compact-miss operators. Adds host-backed HiSparse runtime, index groups, prefill staging, and sparse MLA decode and prefill paths.                                     |
| **Connector and worker data movement** <br> `vllm/distributed/kv_transfer/kv_connector/v1/*`, `vllm/v1/worker/gpu/*`, `vllm/v1/worker/utils.py`                                                | Adds the HiSparse connector and worker. Extends connector lifecycle hooks, transfer results, host mirroring, per-pool zeroing, block-table updates, and block copying.                                                |
| **Model wiring, warmup, and validation** <br> `vllm/model_executor/*`, `vllm/models/*`, `tests/v1/*`, `tests/kernels/*`                                                                        | Threads shared sparse index-group builders through supported models, adds HiSparse warmup, and covers runtime, cache, scheduler, connector, kernel, and end-to-end behavior.                                          |
| **Related transfer and test updates** <br> `vllm/distributed/kv_transfer/kv_connector/v1/nixl/*`, `vllm/distributed/kv_transfer/kv_connector/v1/mooncake/*`, `tests/evals/gsm8k/gsm8k_eval.py` | Updates transfer-group addressing, failure reporting, descriptor registration, shutdown behavior, and the GSM8K request-timeout option.                                                                               |
| **Design documentation** <br> `docs/design/hisparse.md`                                                                                                                                        | Documents HiSparse ownership, metadata flow, residency resolution, spill transactions, runtime responsibilities, and CUDA-only boundaries.                                                                            |

**Estimated code review effort:** 5 (Critical) | \~120 minutes

<!-- final_review_risk_start -->

**Merge Risk:** _🟡 Moderate_ · up to `49d1f`

<!-- final_review_risk_coverage:{"sourceCommitId":"49d1fae38f6f1d6175f8abf49c74f1e0040a92c6","coveredCommitId":"49d1fae38f6f1d6175f8abf49c74f1e0040a92c6","kind":"reviewed"} -->

HiSparse adds host-backed KV residency and spill handling. An unresolved spill-transition accounting concern could cause incorrect resident-page mapping under reclaim pressure, so it should be resolved before merge.

<!-- final_review_risk_end -->

<!-- walkthrough_end -->

<!-- pre_merge_checks_walkthrough_start -->

<details>
<summary>🚥 Pre-merge checks | ✅ 4 | ❌ 1</summary>

### ❌ Failed checks (1 warning)

|     Check name     | Status     | Explanation                                                                                                                                                                                   | Resolution                                                                         |
| :----------------: | :--------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------- |
| Docstring Coverage | ⚠️ Warning | Docstring coverage is 34.12% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 595 functions across 69 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. |

<details>
<summary>✅ Passed checks (4 passed)</summary>

|         Check name         | Status   | Explanation                                                                                                                                                                 |
| :------------------------: | :------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|     Linked Issues check    | ✅ Passed | Check skipped because no linked issues were found for this pull request.                                                                                                    |
| Out of Scope Changes check | ✅ Passed | Check skipped because no linked issues were found for this pull request.                                                                                                    |
|         Title check        | ✅ Passed | The title clearly identifies the main change: HiSparse host-resident sparse-MLA decode hot-buffering.                                                                       |
|      Description check     | ✅ Passed | The description directly explains the HiSparse architecture, host-resident KV storage, GPU hot buffers, residency management, integrations, scope, and performance results. |

</details>

</details>

<!-- pre_merge_checks_walkthrough_end -->

<!-- finishing_touch_checkbox_start -->

<details>
<summary>✨ Finishing Touches 💡 2</summary>

<!-- finishing_touch_suggestion:docstrings -->

<details>
<summary>📝 Generate docstrings 💡</summary>

- [ ] <!-- {"checkboxId":"7962f53c-55bc-4827-bfbf-6a18da830691"} --> Create stacked PR
- [ ] <!-- {"checkboxId":"3e1879ae-f29b-4d0d-8e06-d12b7ba33d98"} --> Commit on current branch

</details>
<!-- finishing_touch_suggestion:resolve_merge_conflict -->
<details open>
<summary>⚔️ Resolve merge conflicts 💡</summary>

- [ ] <!-- {"checkboxId":"c3a5b2e1-4d7f-4a8c-b9d6-e1f2c3d4a5b6"} --> Resolve merge conflict in branch `feat/hisparse-mla-decode-main`

</details>

</details>

<!-- finishing_touch_checkbox_end -->

<!-- tips_start -->

---

Thanks for using [CodeRabbit](https://coderabbit.ai?utm_source=oss\&utm_medium=github\&utm_campaign=vllm-project/vllm\&utm_content=53781)! It’s free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

<details>
<summary>❤️ Share</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A\&url=https%3A//coderabbit.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai)
- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit\&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai\&mini=true\&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit\&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

</details>

<sub>Comment `@coderabbitai help` to get the list of available commands.</sub>

<!-- tips_end -->

**mergify** (2026-09-04T21:55:06Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @MatthewBonanni.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-05T02:18:05Z):

Hi @MatthewBonanni, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**LCAIZJ** (2026-09-06T15:16:02Z):

Setup: 8× H20-3e, CUDA 13.0, torch 2.13
Model: GLM-5.2 (W4A8-CT, GlmMoeDsaForCausalLM, TP=8)

## Three Blocking Issues Found While Testing HiSparse

### 1. Engine initialization crashes with `NotImplementedError` for `tokens_per_state`

With prefix caching enabled, GLM produces a multi-group KV cache configuration containing:

- Approximately 20 resident/hot groups
- An indexer
- A host-source group

`resolve_kv_cache_block_sizes` in `v1/core/kv_cache_utils.py:790` evaluates:



```python
spec.prefix_cacheable
isinstance(spec.tokens_per_state, int)
```

However, `HiSparseHotSpec` and `HiSparseResidentSpec` inherit:



```python
prefix_cacheable = True
```

from the base class without implementing `tokens_per_state`. This causes the engine to crash during startup.

#### Workaround

The following workaround fixed the issue in our environment:



```python
prefix_cacheable = False
```

This is consistent with:

- Their `enable_prefix_caching=False` group construction
- The existing `KpoolTailSpec` behavior

---

### 2. First forward crashes with `"HiSparse GPU slots have no scheduler mapping envelope"`

Our instrumentation shows:

- The scheduler constructs `row_mirrors` correctly.
- On the worker, `state.candidates` is empty when `_resolve_row_mirrors` runs.

#### Root cause

The new base stack defers `start_load_kv` until after the forward pass when:



```python
has_sync_kv_loads = False
```

This is the normal path in:



```text
v1/worker/gpu/kv_connector.py::pre_forward
```

As a result, the connector’s `start_step`, which publishes the mirror-mapping envelope, runs only after the following functions have already attempted to consume it:

- `stage_row_mirror_mapping`
- `finish_forward`

The execution order is effectively:



```text
stage_row_mirror_mapping
  -> finish_forward
  -> start_step publishes the mirror-mapping envelope
```

Therefore, the mapping information is unavailable when the worker needs it.

---

### 3. Host pool becomes full and never recovers, causing an admission livelock

After running 16 concurrent requests of approximately 26K tokens each, with roughly half of their KV cache spilled to the host pool, every new request remains in `WAITING` indefinitely.

At this point:

- EngineCore keeps busy-looping.
- No new request is scheduled.
- The host pool never recovers.

#### Instrumentation results

For one blocked request:



```text
GPU pool:  free 124206 >= need 7518
Host pool: free 261 < need 358
```

The GPU pool has enough capacity, but the host pool does not.

The 5,974-block host pool is entirely occupied by blocks belonging to requests that have already finished. Their hashes were never published.

Replaying the exact same prompt produces:



```text
computed=0
no prefix-cache hit
```

**mergify** (2026-09-06T15:16:31Z):

Hi @MatthewBonanni, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-09-07T03:04:16Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @MatthewBonanni.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-08T17:33:51Z):

Hi @MatthewBonanni, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-09-08T17:43:28Z):

Hi @MatthewBonanni, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**MatthewBonanni** (2026-09-08T17:52:08Z):

/ci run

**github-actions** (2026-09-08T17:52:28Z):

✅ Triggered [Buildkite CI #87746](https://buildkite.com/vllm/ci/builds/87746) for commit `468571f2d2ee`.

**MatthewBonanni** (2026-09-08T20:19:13Z):

/ci run

**github-actions** (2026-09-08T20:19:33Z):

✅ Triggered [Buildkite CI #87768](https://buildkite.com/vllm/ci/builds/87768) for commit `f3d4fa239442`.

**samsja** (2026-09-09T03:09:43Z):

Hey all. I am having some issue running this branch with high concurency, summary of the issue:

We ran GLM-5.3 generating long math reasoning from 32 frozen prompts, with reasoning_effort=max

  - Hardware: 2 nodes × 8 H200s, one aggregated EP16 replica, TP1.
  - Concurrency: 1,024 simultaneous requests—64 per GPU, held constant throughout.
  - Failure onset: GPU KV filled around 13.2k context tokens per request; corruption appeared around 13.9k, about 20 minutes into generation.

So the issue arrive when we increase concurenc it seems to be related to  when GPU KV fills and host fetching begins.

astra as this suggestion:



```
We suspect full CUDA graph replay skips the Python setup for eager host mirroring: each step clears the mirror metadata, replay leaves it unset, and the spill path assumes the host copy is current—potentially exposing stale or unwritten KV after eviction.
```

**MatthewBonanni** (2026-09-09T18:16:03Z):

@samsja thanks for pointing this out! I believe this resulted from some of the rework we did for the blog post. It should be fixed now 👍

**samsja** (2026-09-10T01:36:01Z):

> @samsja thanks for pointing this out! I believe this resulted from some of the rework we did for the blog post. It should be fixed now 👍

didn’t realized you guys had fixed it already, I have a pr that also seems to fix it also I have no clue if its correct (mostly agent debugging it, it is avoid the bug tho) [vllm-project/vllm#56158](https://github.com/vllm-project/vllm/pull/56158)

**mergify** (2026-09-10T04:41:22Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @MatthewBonanni.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**MatthewBonanni** (2026-09-11T01:52:42Z):

/ci run

**github-actions** (2026-09-11T01:53:00Z):

✅ Triggered [Buildkite CI #88227](https://buildkite.com/vllm/ci/builds/88227) for commit `960671385f0a`.

**MatthewBonanni** (2026-09-11T03:54:10Z):

/ci run

**github-actions** (2026-09-11T03:54:28Z):

✅ Triggered [Buildkite CI #88242](https://buildkite.com/vllm/ci/builds/88242) for commit `e995b60e3fa6`.

**mergify** (2026-09-11T09:34:51Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @MatthewBonanni.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**MatthewBonanni** (2026-09-11T14:25:40Z):

/ci run

**github-actions** (2026-09-11T14:26:07Z):

✅ Triggered [Buildkite CI #88343](https://buildkite.com/vllm/ci/builds/88343) for commit `e726216e4644`.

**MatthewBonanni** (2026-09-11T16:07:11Z):

/ci run

**github-actions** (2026-09-11T16:07:32Z):

✅ Triggered [Buildkite CI #88363](https://buildkite.com/vllm/ci/builds/88363) for commit `4720d21851d5`.

**MatthewBonanni** (2026-09-11T18:24:48Z):

/ci run

**github-actions** (2026-09-11T18:25:19Z):

✅ Triggered [Buildkite CI #88390](https://buildkite.com/vllm/ci/builds/88390) for commit `bbc841a107c9`.

**MatthewBonanni** (2026-09-11T20:08:38Z):

/ci run

**github-actions** (2026-09-11T20:08:57Z):

✅ Triggered [Buildkite CI #88411](https://buildkite.com/vllm/ci/builds/88411) for commit `37b1c0136768`.

**njhill** (2026-09-15T22:14:44Z):

I feel that this was rushed in too quickly given how invasive it is. In particular the nontrivial changes to the connector interface and corresponding changes to model runner etc.

Could we not have at least separated the kv connector changes into a follow-on PR so that we could try to get the design as simple as possible

