---
title: '56625'
pageLayout: default
slug: thoughts/inference/vllm-prs/56625
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56625.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56625](https://github.com/aarnphm/aarnphm.github.io/issues/56625) \[DSV4.1\] Add encoder cuda graph support for deepseek-v4.1-flash

[vllm-project/vllm#56625](https://github.com/vllm-project/vllm/pull/56625) · Isotr0py · MERGED · created 2026-09-12T17:54:17Z · merged 2026-09-20T13:51:14Z

## Purpose

Enable CUDA graph capture/replay for the DeepSeek-V4.1 vision encoder (ViT +
aligner) via the `SupportsEncoderCudaGraph` protocol, removing per-image eager
encoder overhead (Python loop, per-image kernel launches, RoPE/cu_seqlens
host computation) from the request critical path.

The captured graph packs a batch of images into one varlen run: ViT blocks
attend per image via `cu_seqlens`, and the aligner’s spatial merge becomes a
gather+mask against precomputed indices. Enabled with
`--compilation-config '{"cudagraph_mm_encoder": true}'`.

## Test Plan

### Setup

- Model: `deepseek-ai/DeepSeek-V4.1-Flash` (real weights, FP8 + FP4 experts)
- Hardware: 1 node, 4 x NVIDIA GB200 (SM100), TP=4
- Attention backend: `FLASHMLA_SPARSE_DSV41` (the SM100-default
  `FLASHMLA_MEGA_ATTN_DSV41` is currently broken upstream for this checkpoint:
  `finalize_mega_attn_weights` permutes `wq_b` assuming the pre-repack
  `[out, in]` layout, but the MXFP8 quant finalize has already transposed it
  to MN-major — unrelated to this change)
- Server: `vllm serve ... --max-model-len 32768 --max-num-seqs 64`
- A/B: identical server + client, only `cudagraph_mm_encoder` toggled.
  With the flag on, 9 budget graphs were captured (64..16384 tokens),
  ViT attention backend `FLASH_ATTN`.

### Commands



```bash
# Server (encoder CG off / on)
vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
    --tensor-parallel-size 4 \
    --attention-backend FLASHMLA_SPARSE_DSV41 \
    --max-model-len 32768 --max-num-seqs 64 \
    [--compilation-config '{"cudagraph_mm_encoder": true}']

# Benchmark: VisionArena-Chat, 256 requests, concurrency 32, 32 output tokens
vllm bench serve \
    --backend openai-chat --endpoint /v1/chat/completions \
    --model deepseek-ai/DeepSeek-V4.1-Flash \
    --dataset-name hf --dataset-path lmarena-ai/VisionArena-Chat \
    --num-prompts 256 --max-concurrency 32 \
    --output-len 32 --ignore-eos \
    --percentile-metrics ttft,tpot,itl,e2el

# e2e unit test
pytest tests/models/multimodal/generation/test_vit_cudagraph.py -k deepseek_v41
```

## Test Result

### Benchmark (VisionArena-Chat, 256 reqs, avg \~446 input tokens incl. image)

| Metric                     | Encoder CG off | Encoder CG on | Speedup |
| -------------------------- | -------------- | ------------- | ------- |
| Benchmark duration (s)     | 64.08          | 24.68         | 2.60x   |
| Request throughput (req/s) | 4.00           | 10.37         | 2.60x   |
| Mean TTFT (ms)             | 4467           | 1894          | 2.36x   |
| Median TTFT (ms)           | 4441           | 691           | 6.43x   |
| P99 TTFT (ms)              | 21348          | 4570          | 4.67x   |
| Mean TPOT (ms)             | 114.2          | 38.3          | 2.98x   |
| Mean E2E latency (ms)      | 8007           | 3081          | 2.60x   |
| P99 E2E latency (ms)       | 21620          | 4858          | 4.45x   |

TPOT also improves because eager per-image encoder runs no longer interleave
with (and preempt) decode iterations.

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**mergify** (2026-09-12T19:02:04Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-14T06:50:45Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-16T17:39:46Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**Isotr0py** (2026-09-17T02:04:27Z):

/ci run

**github-actions** (2026-09-17T02:04:44Z):

✅ Triggered [Buildkite CI #89507](https://buildkite.com/vllm/ci/builds/89507) for commit `55ebc47a4d6b`.

**mergify** (2026-09-17T02:10:53Z):

Hi @Isotr0py, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-09-17T02:18:01Z):

Hi @Isotr0py, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**Isotr0py** (2026-09-17T02:43:17Z):

/ci run —allow-stale

**github-actions** (2026-09-17T02:43:35Z):

✅ Triggered [Buildkite CI #89519](https://buildkite.com/vllm/ci/builds/89519) for commit `70e2b804e657`.

⚠️ This PR is 5 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

**mergify** (2026-09-17T02:52:53Z):

Hi @Isotr0py, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**Isotr0py** (2026-09-17T03:11:03Z):

/ci run —allow-stale

**github-actions** (2026-09-17T03:11:23Z):

✅ Triggered [Buildkite CI #89523](https://buildkite.com/vllm/ci/builds/89523) for commit `f480a59143f8`.

⚠️ This PR is 5 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

**mergify** (2026-09-17T03:16:53Z):

Hi @Isotr0py, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**ChuanLi1101** (2026-09-17T04:08:05Z):

This PR modifies vllm/models/deepseek_v41/amd/vl_model.py but isn’t labeled rocm. Please add the rocm label when the AMD path changes so ROCm review/CI aren’t skipped.

**mergify** (2026-09-17T04:28:51Z):

Hi @Isotr0py, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**Isotr0py** (2026-09-17T04:37:50Z):

/ci run

**github-actions** (2026-09-17T04:38:09Z):

✅ Triggered [Buildkite CI #89537](https://buildkite.com/vllm/ci/builds/89537) for commit `a1b86692f786`.

**gty111** (2026-09-19T08:40:16Z):

/ci run

**github-actions** (2026-09-19T08:40:33Z):

❌ This PR is 136 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5740543164 -->

**Isotr0py** (2026-09-20T04:49:12Z):

/ci run —allow-stale

**github-actions** (2026-09-20T04:49:29Z):

✅ Triggered [Buildkite CI #90035](https://buildkite.com/vllm/ci/builds/90035) for commit `ca70b6d21259`.

