---
title: '57874'
pageLayout: default
slug: thoughts/inference/vllm-prs/57874
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57874.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57874](https://github.com/aarnphm/aarnphm.github.io/issues/57874) \[Bugfix\]\[DSV4.1\] Restrict mHC overlap to full CUDA graphs

[vllm-project/vllm#57874](https://github.com/vllm-project/vllm/pull/57874) · WoosukKwon · MERGED · created 2026-09-21T05:00:07Z · merged 2026-09-21T08:17:00Z

## Problem and fix

DSV4.1’s small-batch eager warmup selected overlap, while breakable PIECEWISE capture selected Mega mHC. That could leave Mega mHC’s stream-local barriers uninitialized until capture, causing startup to fail.

Use overlap only during FULL CUDA graph capture, still bounded by the existing 16-token cutoff and hardware requirements. Eager execution/warmup and PIECEWISE execution retain native mHC dispatch (Mega mHC where supported). Eager warmup therefore initializes Mega mHC on the capture stream. FULL replay retains the captured overlap kernels without executing the Python check per token.

The selection checks actual stream capture and excludes PIECEWISE. This works with V2’s existing FULL-capture context, which advertises NONE while the runner explicitly wraps the model in `torch.cuda.graph`. CUDA graph runner/context code, AMD model code, and shared model-state code are unchanged. There is no separate warmup helper or cache. Mega mHC eligibility remains centralized in its implementation module.

## Validation



```bash
.venv/bin/python -m pytest tests/kernels/test_mhc_kernels.py \
  -k decoder_mixes_match_torch -q -rs

.venv/bin/pre-commit run --files \
  vllm/models/deepseek_v41/nvidia/model.py \
  vllm/models/deepseek_v41/nvidia/ops/mhc.py \
  vllm/models/deepseek_v41/nvidia/ops/mega_mhc.py \
  tests/kernels/test_mhc_kernels.py
```

- **20 passed**, 285 deselected on GB200. Decoder/reference parity covers broadcast, pipeline, residual, and Engram entry paths under disabled overlap, eager execution, FULL capture/replay, PIECEWISE, and larger batches. The FULL cases verify overlap is called during capture, never eager warmup.
- Pre-commit passed.
- Fresh-process real-weight GB200 TP4 FULL_AND_PIECEWISE startup/capture passed on all four ranks with capture sizes \[1,2,4,8\]. Both measured exact8192-input/1024-output C1 requests completed, with no monitored inference-time JIT warnings. Mean TPOT 4.553 ms, median ITL 4.385 ms, P99 ITL 10.368 ms. These two requests are a serving smoke check, not a matched performance comparison or a tail-latency improvement claim.
- Earlier revision’s GSM8K 128-example smoke: 126/128 (98.44%) flexible extraction, strict 106/128. Checkpoint revision `dba1be0a40aa45a94ad051997016db3960a90277`, TP4, FlashInfer FP8 KV, MXFP4 sparse indexer, greedy, five-shot, thinking disabled. This predates the FULL-only-overlap revision and is not a new accuracy evaluation of it. Eager small-batch execution now uses the existing native path, so cross-path bitwise equivalence is not claimed.

## Related work and AI assistance

Open-PR checks found no equivalent fix for the overlap/Mega mHC warmup mismatch exposed by [#57603](https://github.com/aarnphm/aarnphm.github.io/issues/57603). [#57843](https://github.com/aarnphm/aarnphm.github.io/issues/57843) warms standalone DeepGEMM prenorm projection specializations; it does not initialize Mega mHC’s stream-local barriers. This PR changes DSV4.1 dispatch policy instead.

OpenAI Codex assisted with implementation and validation. The human submitter reviewed/tested an earlier revision and requested this FULL-only overlap policy; the revised validation above was performed by the agent.

---

## comments

**WoosukKwon** (2026-09-21T05:18:09Z):

/ci run

**github-actions** (2026-09-21T05:18:26Z):

✅ Triggered [Buildkite CI #90145](https://buildkite.com/vllm/ci/builds/90145) for commit `2c1a18bb1721`.

