---
title: '57603'
pageLayout: default
slug: thoughts/inference/vllm-prs/57603
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57603.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57603](https://github.com/aarnphm/aarnphm.github.io/issues/57603) \[Perf\]\[DSV4.1\] Overlap mHC coefficients for small TP batches

[vllm-project/vllm#57603](https://github.com/vllm-project/vllm/pull/57603) · WoosukKwon · MERGED · created 2026-09-18T17:27:25Z · merged 2026-09-21T00:33:17Z

## Purpose

Overlap shifted mHC coefficient generation with attention/FFN on small batches. The decoder keeps one post/pre path: the existing mHC dispatcher
selects overlap or native execution, while the decoder joins the coefficient
stream before consuming its outputs.

`supports_mhc_overlap` in `.ops.mhc` requires SM100/DeepGEMM, no ubatching,
and compatible projection dimensions: hidden width divisible by 64, and
`hc_mult * (hc_mult + 2)` a positive multiple of 8 no greater than 32.
The exact H5120/hc4 restrictions and helper assertion are removed.
Compatible tensor/data/pipeline/expert-parallel configurations, attention backends,
and LoRA configurations use the same token cutoff, without a separate performance
allowlist. Larger batches, draft layers, and piecewise/breakable graphs retain
native execution. The cutoff is generalized from the TP4 measurements below;
those measurements do not establish a gain for every newly enabled configuration.

The cutoff is **16 tokens**, based on same-instance paired full-forward
measurements with matching tuned GEMM tactics (three samples per arm):

| Tokens | Native ms | Overlap ms | Reduction |
| ------ | --------: | ---------: | --------: |
| 1      |     4.763 |      4.692 |     1.49% |
| 6      |     6.204 |      5.929 |     4.42% |
| 8      |     6.644 |      6.360 |     4.28% |
| 12     |     7.408 |      7.292 |     1.57% |
| 16     |     7.966 |      7.877 |     1.12% |
| 24     |     9.153 |      9.365 |    -2.31% |
| 48     |    11.915 |     12.040 |    -1.05% |
| 96     |    15.292 |     15.247 |     0.29% |
| 192    |    18.820 |     18.977 |    -0.84% |
| 252    |    20.040 |     20.230 |    -0.95% |

All samples are retained; the 24/48-token overlap arms contain outliers
(SD 0.390/0.388 ms). Larger shapes show no reliable useful gain, so keep native
execution there. Selection uses the captured token shape, never actual KV
length. Split-kernel warmup is bounded by the same cutoff.

## Validation

Based on main `17e50b9b76`. Latest validation: **79 GPU tests pass**, covering
exact output parity and changed-input CUDA graph replay across `(hidden_size,
hc_mult)` = `(4096, 2)`, `(5120, 4)`, `(7168, 4)`, plus decoder/reference parity
and graph/batch fallbacks. Pre-commit passes.



```bash
.venv/bin/python -m pytest tests/kernels/test_mhc_kernels.py \
  -k 'overlap_preserves_outputs or decoder_mixes_match_torch' -q -rs
```

Only transient input/scratch tensors (`x`, `mix`, `sqr`) need side-stream allocator
recording. The model retains weights, and callers retain coefficient outputs
until joining the side stream; the helper documents that ownership requirement.

Model/performance measurements below are prior results, not reruns on the new
main base or measurements of every newly enabled configuration.

Measured on GB200×4, model revision `dba1be0a40aa45a94ad051997016db3960a90277`,
FlashInfer 0.6.18.post1, TP4, DSpark fixed length 5, and synthetic 100K ±5% KV.
These are fixed-forward GPU latencies; no default-O2 piecewise-serving gain is
claimed. The prior ten-sample native/automatic pair measured 6.206→5.932 ms (4.42%).

The refactored commit’s three-sample same-instance native/automatic comparison
measures **6.210→5.928 ms (4.55%)**, with identical inputs, KV and tuned GEMM
tactics. Automatic selection activates all 40 target layers. Both arms have
finite outputs and zero eager/graph, changed-input and restored-input replay
errors. Replay checks do not assert native/overlap full-model bitwise parity.

Prior matched real-weight GSM8K: 1271→1268/1319 (96.36→96.13%; 7 improve, 10 regress).
DSpark acceptance length 3.870→3.874. Local FP64 and serialized/async checks support
the numerical change; full-model hidden states are rounding-sensitive. The
refactor preserves the evaluated kernel math and launch order, so this accuracy
evaluation is reused. Prior same-instance allocator peaks increased 16 MiB allocated
and 74 MiB reserved per rank; this is not a serving-capacity measurement.

The projection-width limit of 32 follows DeepGEMM’s device-side output-swizzle
constraint. Newly enabled dimensions have correctness coverage, not additional
end-to-end performance measurements.

## Final-head validation (2026-09-21)

Exact head `479b025e45`, GB200×4, pinned model revision above, TP4 **without
DSpark or all-reduce+mHC fusion**, FlashInfer FP8 attention/MXFP4 sparse indexer.
Baseline uses the same head with overlap disabled before loading.

| Metric                         |               Overlap disabled |            Enabled |
| ------------------------------ | -----------------------------: | -----------------: |
| Forward-only, KV8704           | 4.727 /4.730 ms (before/after) |   4.657 ms (−1.5%) |
| Real8K/1K C1 mean TPOT         |                       5.013 ms |   4.992 ms (−0.4%) |
| Median ITL                     |                       5.011 ms |           4.974 ms |
| P99 ITL                        |                       5.228 ms |           6.049 ms |
| Full GSM8K flexible extraction |             1266/1319 (95.98%) | 1271/1319 (96.36%) |

Forward harness: same initialized synthetic weights/KV, five samples ×20
CUDA-graph replays after5s warmup per arm; slowest-rank GPU time, excluding
lm_head/sampling/scheduler/HTTP. All arms pass finite-output and changed-input
replay checks (zero replay error). This is not cross-arm bitwise parity.

Serving: ten exact8192-input/1024-output requests per arm, greedy, C1, two
warmups; FULL_DECODE_ONLY with eager prefill. All requests succeed; candidate
tail bursts in2/10 requests mean this is not a strong serving-speed claim.
GSM8K: lm-eval0.4.12, five-shot chat, greedy, seed123, C8, max output1024;
matching prompts/targets,10 improve and5 regress, strict formatting79.68% both.

**Warmup caveat:** initial overlap-disabled GSM8K reproduced cached FlashInfer
CuTe tactic compilation under strict JIT monitoring. Final arms use identical
startup-only cached-tactic and DeepGEMM split warmup from the existing fix in
[#57643](https://github.com/aarnphm/aarnphm.github.io/issues/57643), plus identical pinned tactics. Zero monitored runtime JIT afterward;
these workarounds are not part of this PR. No claim of stock-stack zero JIT or
coverage of every newly enabled topology/backend. The79 focused GPU tests
above validate the exact head.

## Duplicate-work check

No open PR found for shifted mHC coefficient/sublayer overlap. [#52725](https://github.com/aarnphm/aarnphm.github.io/issues/52725) concerns
XPU norm fusion; [#42735](https://github.com/aarnphm/aarnphm.github.io/issues/42735) changes BF16 shared staging in the older mHC kernel;
[#56761](https://github.com/aarnphm/aarnphm.github.io/issues/56761) changes GLM JIT warmup; [#51802](https://github.com/aarnphm/aarnphm.github.io/issues/51802) registers NVIDIA V4 warmup variants.
None implements this scheduling change.

AI assistance was used to implement, benchmark and test this change.

---

## comments

**WoosukKwon** (2026-09-20T23:26:17Z):

/ci run

**github-actions** (2026-09-20T23:26:34Z):

✅ Triggered [Buildkite CI #90102](https://buildkite.com/vllm/ci/builds/90102) for commit `479b025e45cb`.

