---
title: '55960'
pageLayout: default
slug: thoughts/inference/vllm-prs/55960
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/55960.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#55960](https://github.com/aarnphm/aarnphm.github.io/issues/55960) \[Perf\] Add fused DFlash2 grouped convolution

[vllm-project/vllm#55960](https://github.com/vllm-project/vllm/pull/55960) · mgoin · MERGED · created 2026-09-08T19:19:20Z · merged 2026-09-17T21:39:00Z

## Summary

- Fuse DFlash2’s grouped dynamic convolution in one Triton custom op, keeping the existing Torch CPU fallback. The kernel stays in `qwen3_dflash2.py`.
- Use row-aligned tiles: 1024 elements for at least 128 rows with hidden width divisible by 1024, otherwise 512. Four warps; no autotuner.
- Cover non-power-of-two dimensions, strided coefficient views, multiple tap counts, and both tile paths in FP32/BF16.
- Add real-checkpoint GSM8K accuracy and acceptance-length coverage to the existing H200 35 GB DFlash nightly job. Keep this case excluded from its AMD mirror; the Speculators/MTP job is unchanged.

The implementation docstring credits LightSeek TokenSpeed PR [lightseekorg/tokenspeed#1399](https://github.com/lightseekorg/tokenspeed/pull/1399).

## End-to-end validation

One NVIDIA B300, TP=1, V2 model runner, compilation and CUDA graphs enabled. Target: `nvidia/Qwen3.8-27B-NVFP4`; unchanged BF16 draft: `z-lab/Qwen3.8-27B-DFlash2`. Full GSM8K: 1,319 questions, 5-shot chat, thinking disabled, temperature 0, max output 256 tokens, max model length 4,096, max batch 32, seven speculative tokens.

| Implementation                                         | GSM8K accuracy | Mean acceptance length |
| ------------------------------------------------------ | -------------- | ---------------------- |
| Original Torch convolution, NVFP4 target, run 1        | 92.0%          | 6.14                   |
| Original Torch convolution, NVFP4 target, repeat       | 91.2%          | 6.13                   |
| Fused kernel before final tuning, NVFP4 target, run 1  | 91.1%          | 6.13                   |
| Fused kernel before final tuning, NVFP4 target, repeat | 92.2%          | 6.14                   |
| Final row-aligned kernel, NVFP4 target                 | **92.2%**      | **6.14**               |

All passed. Acceptance length is `1 + accepted_tokens / draft_steps`. Accuracy varies across runs; this is not an exact-output-equivalence claim. The compiled draft graph was inspected to confirm it invokes the convolution custom op. CI checks accuracy >= 90% and AL >= `6.14 * 0.95` (5.833).

The NVFP4 target reduces reported target-plus-draft model-load memory from 54.0 GiB to 22.85 GiB; this is not total serving/KV-cache memory. The case runs in the existing H200 35 GB DFlash nightly job, with its 90-minute timeout unchanged, and remains excluded from the AMD mirror. H200 has a Marlin weight-only fallback for the NVFP4 target. Local validation was on B300; H200 execution and memory fit remain unverified.

Earlier BF16-target validation also passed: original Torch convolution 92.2% / AL 6.19; initial fused kernel 92.0% / AL 6.19 in two runs. CI now uses the smaller NVFP4 checkpoint instead.

## Kernel performance

### Initial fusion versus Torch (before the final row-aligned tuning)

Actual checkpoint shape: hidden size 5,120, BF16, block size 8, group size 16, two taps, strided coefficients. B300, FlashInfer CUPTI timing, CUDA graphs, cold L2, four alternating-order trials; median latency. Baseline is the existing expression under `torch.compile`. Correctness checked before timing.

| Batch | Torch / Triton latency (us) | Torch / Triton effective GB/s | Speedup |
| ----- | --------------------------- | ----------------------------- | ------- |
| 1     | 1.889 / 1.920               | 103 / 101                     | 0.98x   |
| 2     | 2.064 / 2.016               | 179 / 183                     | 1.02x   |
| 4     | 2.176 / 2.112               | 329 / 339                     | 1.03x   |
| 8     | 2.496 / 2.320               | 566 / 609                     | 1.08x   |
| 16    | 3.296 / 2.720               | 851 / 1,032                   | 1.21x   |
| 32    | 4.576 / 3.599               | 1,222 / 1,554                 | 1.27x   |

Effective bandwidth counts each input once and the output once: `2 * (2*rows*5120 + rows*2*320 + 2*5120)` bytes, `rows = batch*8`. These are kernel-only gains, not end-to-end speedups; batch 1 is approximately neutral.

### Final row-aligned tuning versus the posted fused kernel

Integrated confirmation on all 11 surveyed hidden-width/block-size pairs, at batches 1/32/256, BF16, group 16, two taps, strided coefficients. B300, CUPTI, CUDA graphs, cold L2, four alternating-order trials of 100 ms. Compared with posted commit `fd19503785`: batch 1 is effectively unchanged; batch 32 is 1.006–1.172x faster; batch 256 is 1.078–1.328x faster.

Representative batch-256 results (posted fused kernel → final tuned kernel):

| Hidden / block | Tile | Useful GB/s | Latency (us)    | Speedup |
| -------------- | ---- | ----------- | --------------- | ------- |
| 2560 / 8       | 512  | 2649 → 2856 | 8.384 → 7.776   | 1.078x  |
| 5120 / 8       | 1024 | 3038 → 3921 | 14.624 → 11.328 | 1.291x  |
| 6656 / 16      | 512  | 3300 → 3610 | 35.057 → 32.049 | 1.094x  |

Useful bytes = `2 * (2*rows*hidden + (2*rows-batch)*(hidden/16) + 2*hidden)`, excluding unused second-tap coefficients at block starts. This is an ideal useful-bandwidth estimate, not measured HBM traffic. These remain kernel-only speedups. Block-7/anchor/MoE entries in the shape sweep are kernel coverage, not full-model compatibility claims.

## Tests

All GPU workloads used `chg run -g=1 --`; no `CUDA_VISIBLE_DEVICES` overrides. Commands below assume the checkout’s Python environment is activated (including its `bin` directory on PATH):



```bash
chg run -g=1 -- .venv/bin/python -m pytest tests/v1/spec_decode/test_dflash2.py -q
chg run -g=1 -- env VLLM_WORKER_MULTIPROC_METHOD=spawn .venv/bin/python -m pytest -v -s tests/v1/e2e/spec_decode/acceptance_rates/dflash/test_dflash.py -k dflash2
pre-commit run --files .buildkite/test_areas/spec_decode.yaml tests/v1/e2e/spec_decode/acceptance_rates/dflash/test_dflash.py tests/v1/spec_decode/test_dflash2.py vllm/model_executor/models/qwen3_dflash2.py
```

- Focused suite: **16 passed**, including the aligned 1024-tile path and non-aligned 512-tile fallback.
- Integrated scratch boundary sweep: **570/570 bitwise comparisons passed** against the posted kernel. Covers all 11 model shape pairs, FP32/FP16/BF16, empty/one-row inputs, strided inputs/coefficients, ragged rows/widths, 1/2/3 taps, log-scaled values, and 127/128/129-row boundaries.
- Full NVFP4 model test: **1 passed** for each of the five runs above, including the final tuned implementation (**226.52 s**, including startup). First NVFP4 runs took about 5.5 minutes due to compilation/autotuning; warm-cache runs took 3.5–3.7 minutes. The smaller checkpoint is not a claim of faster CI.
- Changed-file pre-commit: **passed**, including mypy, SPDX, forbidden-import and CUDA API checks. CI YAML parsing/selection checks passed.
- Local environment: PyTorch 2.13.0, Triton 3.7.1, Transformers 5.15.0, FlashInfer 0.6.17. Initial startup failed on both branches due to a missing generated FlashMLA Python module in the worktrees; linking the existing installed artifact fixed the environment without code changes. The first NVFP4 attempt on both branches also stopped before evaluation because `ninja` was missing from PATH; using the already-installed venv `bin` directory resolved it without dependency/code changes.

## Duplicate work

Rechecked open vLLM PRs for `dflash2 conv`, `fused grouped conv`, and DFlash2 GSM8K/acceptance coverage. No other open PR implements this kernel/dispatch or this CI case. The companion TP-reduction and packed-top-k PRs remain separate.

AI assistance was used to analyze the upstream implementation, implement the change, and write/run the tests and benchmarks. Human submitter review is required before merge.

---

## comments

**mgoin** (2026-09-13T19:23:22Z):

/ci run

**github-actions** (2026-09-13T19:23:39Z):

✅ Triggered [Buildkite CI #88676](https://buildkite.com/vllm/ci/builds/88676) for commit `6d74637eab89`.

**mgoin** (2026-09-17T20:03:08Z):

/ci run

**github-actions** (2026-09-17T20:03:26Z):

✅ Triggered [Buildkite CI #89703](https://buildkite.com/vllm/ci/builds/89703) for commit `7277fa55bdc9`.

