---
title: '57632'
pageLayout: default
slug: thoughts/inference/vllm-prs/57632
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57632.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57632](https://github.com/aarnphm/aarnphm.github.io/issues/57632) \[DFlash\] Capture the context K/V precompute in the draft CUDA graph

[vllm-project/vllm#57632](https://github.com/vllm-project/vllm/pull/57632) · Juntian777 · MERGED · created 2026-09-18T21:17:51Z · merged 2026-09-24T21:38:04Z

## Purpose

Even when the query forward replays a FULL CUDA graph, the DFlash/DSpark draft step still runs the context K/V precompute (`precompute_and_store_context_kv`) eagerly outside the graph, because the number of context tokens varies from step to step.

This PR captures the context K/V precompute ahead of the query forward in the draft’s FULL decode graph. Each graph stores one full verify of context rows per request; rows past the batch are padded with `PAD_SLOT_ID` so they write no K/V, and any prefill context beyond the graph’s rows is stored eagerly before replay. The precompute has no host syncs, so every DFlash drafter captures it. The draft attention metadata build stays eager, as before.

## Validation

Capturing the context K/V precompute must not change the speculator’s proposals. The checks below therefore compare acceptance and accuracy between this branch’s parent commit (`c723a831a8`) and `d8de1d3a2f`, and serving performance against current main, with everything else held fixed.

### Setup

- **Hardware / software:** 4× GB200, FlashInfer 0.6.18.post1, `VLLM_USE_V2_MODEL_RUNNER=1`.
- **Model:** DeepSeek-V4.1-Flash (`dba1be0a40aa45a94ad051997016db3960a90277`).
- **Speculative decoding:** DSpark with 5 speculative tokens and full decode CUDA graphs.
- **Topologies:**
  - **DEP4:** MegaMoE backend with adaptive verification.
  - **TP4:** auto MoE backend with fixed verification.

### Acceptance and GSM8K accuracy

Each pair runs the full GSM8K test set (1,319 identical 5-shot completion prompts) with greedy decoding, concurrency 32, a 1,024-token output cap, and strict answer extraction.

Acceptance length is read from the engine’s `vllm:spec_decode_*` counters over the whole run for DEP4. For TP4, it is aggregated from the engine’s periodic SpecDecoding log windows over the steady state; the same method is used on both sides of each pair.

| Configuration | Acceptance length, incl. bonus (parent → PR) |                  GSM8K (parent → PR) |
| ------------- | -------------------------------------------: | -----------------------------------: |
| DEP4          |                            3.411 → **3.434** | 1234 → 1228 / 1319 (93.56% → 93.10%) |
| TP4           |                            3.471 → **3.463** | 1227 → 1219 / 1319 (93.03% → 92.42%) |

- The acceptance deltas are within ±1% and have opposite signs across the two topologies, and the per-position acceptance curves coincide. The captured precompute therefore does not change what the drafter proposes.
- The GSM8K deltas are run-to-run noise: two repeats of the same DEP4 parent configuration scored 0.9219 and 0.9356 (an 18/1319 spread), which is wider than any delta in the table.
- These results establish parity with the parent; they do not claim an accuracy improvement.

<details>
<summary>GSM8K server configuration</summary>

Common flags:



```bash
--reasoning-parser deepseek_v41 \
--attention-config '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' \
--kv-cache-dtype fp8 --max-model-len 8192 --max-num-seqs 128
```

DSpark is configured with 5 speculative tokens.

- **DEP4:** `--data-parallel-size 4 --enable-expert-parallel --kernel-config '{"moe_backend":"deep_gemm_mega_moe"}'`, with adaptive verification enabled.
- **TP4:** `--tensor-parallel-size 4`, with the auto MoE backend and adaptive verification disabled.

</details>

### Serving benchmark

TP4, DSpark (5 speculative tokens, synthetic acceptance length 5.5), random dataset with 128 input / 1,024 output tokens. **Base** is main and **PR** is `d8de1d3a2f`. CPU/step is the host time per engine step, measured with an injected per-step probe.

| Concurrency |       CPU/step |                    TPOT |                      ITL |
| ----------: | -------------: | ----------------------: | -----------------------: |
|           1 | 6.77 → 5.03 ms | 1.27 → 1.08 ms (−15.2%) |  6.95 → 5.88 ms (−15.3%) |
|           8 | 6.64 → 5.24 ms |  1.87 → 1.84 ms (−1.6%) |   9.96 → 9.90 ms (−0.6%) |
|          64 | 6.82 → 5.43 ms |  3.90 → 3.87 ms (−0.7%) | 19.97 → 20.01 ms (+0.2%) |

The PR cuts host time per step by 1.4–1.7 ms. At concurrency 1 the step is host-bound, so this carries through to a \~15% lower TPOT and ITL; at higher concurrency the GPU dominates and latency is roughly unchanged.

<details>
<summary>Serving benchmark configuration</summary>



```bash
vllm serve <model> --tensor-parallel-size 4 \
  --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' \
  --kv-cache-dtype fp8 --no-enable-prefix-caching \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"rejection_sample_method":"synthetic","synthetic_acceptance_length":5.5}'

vllm bench serve --dataset-name random --random-input-len 128 --random-output-len 1024 \
  --ignore-eos --max-concurrency <1|8|64>
```

</details>

### Tests



```bash
.venv/bin/python -m pytest tests/v1/attention/test_dspark_noncausal_sparse_mla.py -q
```

## AI assistance

AI assistance was used for this change: the implementation, the evaluation runs, and this description were produced with Claude Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**mergify** (2026-09-18T23:39:10Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Juntian777.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-22T02:11:21Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Juntian777.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-23T10:22:46Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Juntian777.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**Juntian777** (2026-09-23T21:36:37Z):

> Thanks for updating the PR, this looks much cleaner. After thinking about it for some time, I’d prefer if we only ship the `precompute_context_kv` fusion into the cudagraph, and not the `build_attn()`. I think that the ability to fuse the attention metadata build into the cudagraph should be a property of the attention backends, not the model.
>
> We already have a pattern for fusing metadata builds into the cudagraph. It’s done here for MTP/EAGLE:
>
> <https://github.com/vllm-project/vllm/blob/711fc55c10a2bbbefcb12456c43ba5eca01be3f6/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py#L637-L638>
>
> Basically, we enable fusion whenever that method is implemented by all of the draft model’s attention backends. It’s intended to be lightweight, cudagraph-capture safe, and only for decodes. It enables drafting the last N-1 tokens without needing to rebuild the metadata each time.
> I’d like to bring that pattern here for DFlash/DSpark, but as a separate effort. Some extra changes are needed for some of DSV4.1’s backends to enable this. I’m planning to work on this soon, but if the optimization is needed urgently I’m happy to share the details of what’s needed.
>
> In summary, can you please strip the attention metadata build from the cudagraph, and just fuse `precompute_context_kv`? That should simplify things further, while still keeping most of the performance gains.

@TheEpicDolphin Thanks for the advice. I agree that attention metadata cudagraph fusion should live in the attention backend. I’ve updated the PR to keep only precompute_context_kv in the cudagraph now and removed build_attn(). I’ve also updated the benchmark results accordingly.

**github-actions** (2026-09-23T22:56:57Z):

✅ @Juntian777, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**Juntian777** (2026-09-24T00:23:07Z):

/ci run

**github-actions** (2026-09-24T00:23:25Z):

✅ Triggered [Buildkite CI #90817](https://buildkite.com/vllm/ci/builds/90817) for commit `f2b00d1f989e`.

