---
title: '49718'
pageLayout: default
slug: thoughts/inference/vllm-prs/49718
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/49718.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#49718](https://github.com/aarnphm/aarnphm.github.io/issues/49718) \[Attention\] Add FlashInfer XQA decode support on SM12x

[vllm-project/vllm#49718](https://github.com/vllm-project/vllm/pull/49718) · askliar · MERGED · created 2026-07-24T12:14:19Z · merged 2026-08-11T23:32:49Z

## Summary

Enable FlashInfer XQA decode on SM120/SM121 through FlashInfer’s dedicated XQA API.

- Select XQA for SM12x decode.
- Support uniform and ragged speculative decode through `q_len_per_req` and `q_cu_seq_lens`.
- Build the packed causal or non-causal draft mask required by XQA.
- Support sliding-window decode, attention sinks, non-causal draft attention, and uniform-batch CUDA graphs on SM12x.
- Keep the existing SM90 shared-XQA path and SM100 trtllm-gen path unchanged.
- Align the Python, cubin, JIT-cache, and Docker pins on FlashInfer 0.6.16.post1.

## Dependency

This PR requires [FlashInfer 0.6.16.post1](https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.6.16.post1), which includes the required upstream support from [flashinfer#4137](https://github.com/flashinfer-ai/flashinfer/pull/4137) and [flashinfer#4199](https://github.com/flashinfer-ai/flashinfer/pull/4199). The Python, cubin, and CUDA-specific JIT-cache artifacts are available in the release.

## Scope

This is not a duplicate of [#47306](https://github.com/aarnphm/aarnphm.github.io/issues/47306) or [#37592](https://github.com/aarnphm/aarnphm.github.io/issues/37592). Those drafts target the legacy shared decode API and older FlashInfer/base revisions; this PR is limited to SM12x and targets the dedicated 0.6.16.post1 XQA API.

SM90 enhancements and SM100 changes are intentionally excluded and will be handled separately.

SM12x NVFP4 KV-cache support is also out of scope and should build on this PR through [#49818](https://github.com/aarnphm/aarnphm.github.io/issues/49818).

## Validation



```bash
pre-commit run --files docker/Dockerfile docker/versions.json requirements/cuda.txt tests/kernels/attention/test_use_trtllm_attention.py tests/v1/attention/test_attention_backends.py vllm/utils/flashinfer.py vllm/v1/attention/backends/flashinfer.py

.venv/bin/python -m pytest tests/kernels/attention/test_use_trtllm_attention.py -q
.venv/bin/python -m pytest tests/v1/attention/test_attention_backends.py::test_flashinfer_xqa_draft_masks tests/v1/attention/test_attention_backends.py::test_flashinfer_xqa_query_lens_preserve_cudagraph_padding tests/v1/attention/test_attention_backends.py::test_flashinfer_xqa_query_lens_require_exact_uniform_product tests/v1/attention/test_attention_backends.py::test_flashinfer_xqa_decode_correctness -q
```

- Pre-commit passed, including Ruff, formatting, mypy, dependency compilation, and Docker version consistency.
- The focused pytest commands could not run in the current checkout because its local `.venv` is absent; no local CUDA runtime is available.
- SM121 validation confirmed `decode_backend=xqa` and GSM8K at concurrency 64: 0.8764 flexible / 0.8749 strict. A concurrency-1 run exposed the CUDA-graph padding mismatch addressed by the current `q_cu_seq_lens` handling; that fix still needs a post-change GPU rerun.

AI assistance was used to prepare this pull request.

---

## comments

**mergify** (2026-07-25T22:09:01Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**seanyourhighness** (2026-07-29T22:15:34Z):

## SM12x NVFP4 integration note

This PR’s SM12x XQA routing and the NVFP4 enablement in [#49818](https://github.com/aarnphm/aarnphm.github.io/issues/49818) are complementary, but they overlap in `vllm/utils/flashinfer.py` and `vllm/v1/attention/backends/flashinfer.py` and will need an explicit integration order.

The current revision here enables XQA for SM90/SM12x only when `kv_cache_dtype.startswith("fp8")`. Its NVFP4 initialization still requires both TRTLLM prefill and decode support, and the wrapper path still treats NVFP4 as trtllm-gen-only. Therefore this PR does **not** enable SM120 NVFP4 KV.

The accompanying claim that XQA does not support NVFP4 KV needs narrowing. FlashInfer 0.6.15’s XQA implementation has an SM120 NVFP4-KV path. [#49818](https://github.com/aarnphm/aarnphm.github.io/issues/49818) carries the additional vLLM-side pieces required to reach it safely:

- the SM120 NVFP4 dtype/dispatch gate;
- FA2 prefill plus XQA decode;
- model-dtype Q/O and FP8-output-buffer handling;
- the SM120 5-D NVFP4 layout fix for the [#44455](https://github.com/aarnphm/aarnphm.github.io/issues/44455) regression.

That layout point is load-bearing on RTX 5090: current-main gates without it produced token-id-0 collapse 5/5, while changing only the NVFP4 layout produced 5/5 correct; the resulting branch also passed 8/8 concurrency with a 13,158-token needle. The fuller masked-XQA validation behind [#49010](https://github.com/aarnphm/aarnphm.github.io/issues/49010) passed 3/3 cold and 3/3 cached \~31k needles under true FULL MTP, with mask rows `1, 3, 7, 15` for q_len=4.

Suggested split: keep this PR as the generic XQA API/mask/cudagraph foundation, then rebase/compose [#49818](https://github.com/aarnphm/aarnphm.github.io/issues/49818)’s distinct SM120 NVFP4 contract and layout changes on top. I can run the integrated revision on RTX 5090 once the required FlashInfer 0.6.16 artifacts are available. Related root-cause trail: [#49010](https://github.com/aarnphm/aarnphm.github.io/issues/49010), [#49011](https://github.com/aarnphm/aarnphm.github.io/issues/49011), [#49818](https://github.com/aarnphm/aarnphm.github.io/issues/49818).

<sub>Disclosure per AGENTS.md: this diff audit and comment were AI-assisted. The cited hardware runs and measurements were produced and verified by me.</sub>

**mergify** (2026-07-30T13:19:35Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-08-09T10:06:31Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**askliar** (2026-08-10T22:26:52Z):

@pavanimajety @mgoin could you, please, take a look? FI0.6.16 has been updated so now this PR is fully compatible.

**pavanimajety** (2026-08-10T22:49:34Z):

/ci run

**github-actions** (2026-08-10T22:50:08Z):

✅ Triggered [Buildkite CI #83226](https://buildkite.com/vllm/ci/builds/83226) for commit `3d1eac09afd5`.

**Suppressor72** (2026-08-28T21:27:12Z):

Thanks for the XQA + ragged q_cu foundation — we built conditional device/CPU query-length-mismatch support for adaptive verification directly on top of it (SM120, validated locally): details and measurements in our comment on [#51303](https://github.com/aarnphm/aarnphm.github.io/issues/51303), [branch here](https://github.com/Suppressor72/vllm/tree/feat/flashinfer-adaptive-mismatch).

