---
title: '56061'
pageLayout: default
slug: thoughts/inference/vllm-prs/56061
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56061.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56061](https://github.com/aarnphm/aarnphm.github.io/issues/56061) \[4/N\] Expose HiSparse cache metrics via KV connector stats

[vllm-project/vllm#56061](https://github.com/vllm-project/vllm/pull/56061) · NickLucche · MERGED · created 2026-09-09T10:50:44Z · merged 2026-09-12T17:47:56Z

Alternative to [#53782](https://github.com/aarnphm/aarnphm.github.io/issues/53782), reporting the same HiSparse hot-buffer stats through the existing `KVConnectorStats` pipeline instead of adding a new `finish_step` connector hook.

## What stays identical

- Kernel counters (`atomicAdd` in the CUDA kernels) and the swap-ordering changes in `runtime.py` are unchanged from the original PR.
- Same 2000-step async sampling interval with a CUDA event for D2H ordering.
- Same three Prometheus counters: `vllm:hisparse_hits`, `vllm:hisparse_lookups`, `vllm:hisparse_l3_lookups`.

## What changes

Instead of a new `finish_step` hook on `KVConnectorBase` (plus edits to scheduler, outputs, aggregator, loggers, and `stats.py`), the worker now exposes the snapshots through the existing hook:



```python
def get_kv_connector_stats(self) -> Optional[HiSparseKVConnectorStats]: ...
```

and the connector wires it into the standard pipeline:



```
worker.get_kv_connector_stats()          # HiSparseKVConnectorStats (list of snapshot deltas)
  -> scheduler picks it up in SchedulerStats
  -> build_kv_connector_stats()          # HiSparseKVConnectorStats.from_proxy_stats
  -> HiSparsePromMetrics.observe()       # vllm:hisparse_* counters
```

This is the same contract NIXL, the offloading connector, and mooncake already follow — transfer/cache telemetry goes through `KVConnectorStats` and the connector’s `build_prom_metrics`, with zero core plumbing. HiSparse’s hot-buffer residency counters are connector telemetry, so they fit the same slot the offloading connector uses for its allocation-failure/lookup-delay stats.

Net effect: the diff drops from 16 files to 9 and avoids changes to `base.py`, `model_runner.py`, `outputs.py`, scheduler, loggers, and core `stats.py`.

Also picks up two small cleanups over the original:

- Stats emission is gated on the connector being configured (the connector owns its stats, rather than gating on `attention_config` in the model runner).
- The sampling interval advances even on no-forward steps, so long prefill stretches don’t stretch the sampling cadence.

Because the stats flow through `KVConnectorStats`, CLI logging comes for free:



```
INFO ... Cumulative HiSparse stats: hits=12 lookups=4 l3_lookups=64
```

## Validation



```text
.venv/bin/python -m pytest tests/v1/kv_connector/unit/test_hisparse_stats.py tests/v1/worker/test_utils.py::test_hisparse_worker_get_kv_connector_stats_reads_completed_snapshot -q
6 passed

PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pre_commit run --files tests/v1/attention/test_sparse_mla_backends.py
All applicable hooks passed, including Ruff, formatting, and mypy.
```

Model evaluation is not applicable because this change only reports telemetry and does not affect model outputs.

Duplicate check: this remains the `KVConnectorStats`-based alternative to [#53782](https://github.com/aarnphm/aarnphm.github.io/issues/53782); searches for open PRs mentioning [#56061](https://github.com/aarnphm/aarnphm.github.io/issues/56061) and HiSparse metrics found no duplicate of this implementation.

AI assistance: OpenAI Codex helped audit CI and prepare the follow-up cleanup. Human review and ownership remain with the submitter.

---

## comments

**NickLucche** (2026-09-09T10:52:47Z):

/ci run

**github-actions** (2026-09-09T10:53:05Z):

✅ Triggered [Buildkite CI #87897](https://buildkite.com/vllm/ci/builds/87897) for commit `0e1329303c8b`.

**mergify** (2026-09-10T06:30:26Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @NickLucche.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**LucasWilkinson** (2026-09-12T04:44:30Z):

Retargeted to `main` after the previous base branch was merged and deleted. Leaving this closed until the head branch is rebased onto `main`, since the previous base was squash-merged.

**mergify** (2026-09-12T04:48:30Z):

Documentation preview: [https://vllm—56061.org.readthedocs.build/en/56061/](https://vllm--56061.org.readthedocs.build/en/56061/)

**LucasWilkinson** (2026-09-12T04:54:35Z):

/ci run

**github-actions** (2026-09-12T04:54:54Z):

✅ Triggered [Buildkite CI #88476](https://buildkite.com/vllm/ci/builds/88476) for commit `b6dc892489e1`.

**LucasWilkinson** (2026-09-12T15:36:04Z):

/ci run

**github-actions** (2026-09-12T15:36:22Z):

✅ Triggered [Buildkite CI #88527](https://buildkite.com/vllm/ci/builds/88527) for commit `a748d5636562`.

