---
title: '53896'
pageLayout: default
tags:
  - issuecomment-5427847488
slug: thoughts/inference/vllm-prs/53896
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/53896.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#53896](https://github.com/aarnphm/aarnphm.github.io/issues/53896) \[Model\] Support Qwen3.8-Flash-Next

[vllm-project/vllm#53896](https://github.com/vllm-project/vllm/pull/53896) · peakcrosser7 · MERGED · created 2026-08-26T12:56:50Z · merged 2026-08-31T05:57:56Z

## Purpose

support <https://huggingface.co/Qwen/Qwen3.8-Flash-Next>

## How to run



```
vllm serve Qwen/Qwen3.8-Flash-Next
          --served-model-name qwen3.8-flash-next
          -tp 4
          --enable-prefix-caching
          --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'
```

Enable PLE offload(note PLE offload support is in [vllm-project/vllm#53899](https://github.com/vllm-project/vllm/pull/53899))



```
VLLM_PLE_CPU_OFFLOAD=1
```

## Validation

- Without offload:
  - Weight formats: BF16, FP8, and NVFP4
  - Platforms: GB300, GB200, H200, and MI355X (BF16 only)
  - Parallel configurations: TP2, TP4, TEP4

- With N-gram embedding offload:
  - Weight formats: BF16 and FP8 (NVFP4 is not currently supported)
  - Platform: GB200
  - Parallel configurations: TP2, TP4, and DEP4

The validation results were provided by Inferact. See the [vLLM recipe](https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next) for details.

**Note:** This PR is being updated with the latest changes from `main` and may be unstable during review. Please use PR [#53899](https://github.com/aarnphm/aarnphm.github.io/issues/53899) for builds in the meantime.

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

**BEFORE SUBMITTING, PLEASE READ <https://docs.vllm.ai/en/latest/contributing>** (anything written below this line will be removed by GitHub Actions)

---

## comments

**mergify** (2026-08-26T12:57:41Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @peakcrosser7.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**github-actions** (2026-08-26T12:58:26Z):

✅ @peakcrosser7, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**github-actions** (2026-08-26T14:44:50Z):

❌ @oops-oom, Only reviewers with write access can use CI commands before CI is delegated to the PR author.

<!-- vllm-ci-command:5427022164 -->

**mergify** (2026-08-26T15:03:11Z):

Hi @peakcrosser7, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**nichdiekuh** (2026-08-26T15:28:30Z):

Thanks for this PR! Sharing a data point from the `release/qwen38next` branch image (`vllm/vllm-openai:qwen38-flash-next`) on **2× RTX PRO 6000 Blackwell (SM120), TP=2**, with the **NVFP4** checkpoint [`RadixArk/Qwen3.8-Flash-Next-NVFP4`](https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4) and `VLLM_PLE_CPU_OFFLOAD=1`.

Good news: the architecture is recognized and **weights load cleanly** (38.47 GiB/rank, 206 shards, TP=2). The PLE offload connector registers the n-gram embedding layer:



```
PleOffload: registered 1 PleOffloadLayer(s) ... ['language_model.model.layers.1.ple.ple_embedding']
```

…but startup then fails in the PLE offload worker:



```
RuntimeError: PLE offload worker failed during startup:
ValueError("There is no module or parameter named 'ngram_embedding.weight_scale'
in Qwen3_8FlashNextNGramEmbedding. The available parameters belonging to
ngram_embedding (VocabParallelEmbedding) are: {'ngram_embedding.weight'}")
  File ".../vllm/v1/worker/gpu_worker.py", line 294, in wait_ple_offload_ready
  File ".../vllm/v1/ple_offload/worker.py", line 227, in wait_for_ready
```

**Root cause (hypothesis):** this NVFP4 checkpoint leaves the n-gram embedding **un-quantized** (only `ngram_embedding.weight`, no `weight_scale`), but the PLE CPU-offload path appears to assume every offloaded module carries quant scale params — so the offload worker looks up `weight_scale` on an un-quantized `VocabParallelEmbedding` and raises.

Notably this is **not** an SM120 kernel or attention issue — attention is never reached, the arch loads fine. It looks like a quant × PLE-offload interaction: the offload worker likely needs to tolerate modules that only expose `weight` (no `weight_scale`).

Happy to share the full boot log or test a patch on this hardware.

**peakcrosser7** (2026-08-26T15:37:48Z):

Hi @nichdiekuh! Thanks for your feedback!
PLE-Offload for NVFP4 weights isn’t fully supported yet due to time constraints. We’ll get it fixed in a PR shortly. Please try BF16 / FP8 weight PLE-Offload for now.

**OsakaTX** (2026-08-26T16:10:11Z):

Tested this branch on 2x DGX Spark (GB10 / sm_121a, aarch64) with TP=2 against the community quant `RadixArk/Qwen3.8-Flash-Next-NVFP4`. It loads and serves correctly after one small change; sharing in case it is useful.

**Gap: hybrid-quant checkpoints never activate the FP8 PLE path.**

`_get_ple_embedding_quant_method()` in `vllm/models/qwen4_exp/nvidia/ple_layer.py` gates on `isinstance(quant_config, Fp8Config)`. The RadixArk checkpoint is hybrid: modelopt NVFP4 for the routed experts, but the PLE table is stored as FP8 shards (`model-plefp8-*.safetensors`) with a single global `ngram_embedding.weight_scale`. Since the model-level quant config is ModelOpt rather than Fp8Config, the gate returns None, the PLE is built as an unquantized `VocabParallelEmbedding`, and loading fails with:



```
ValueError: There is no module or parameter named 'ngram_embedding.weight_scale' in Qwen4ExpNGramEmbedding. The available parameters belonging to ngram_embedding (VocabParallelEmbedding) are: {'ngram_embedding.weight'}
```

Workaround that works: unconditionally return `Qwen4ExpPLEFp8EmbeddingMethod()` for the PLE (I used an env-var gate for testing). A cleaner fix might detect FP8 PLE shards from the checkpoint itself, or extend the gate to accept ModelOpt-family configs whose checkpoints carry FP8 PLE shards. Happy to test a revision on this hardware.

**Results with that change** (2x GB10, TP=2 over RoCE, `--enforce-eager`, `--max-model-len 65536`, no speculative decoding):

- weights load clean, 61.4 GiB per rank
- coherent generation, temp-0 sanity checks pass
- tool calling works with `--tool-call-parser qwen3_coder`
- reasoning extraction works with `--reasoning-parser qwen3`
- roughly 16 tok/s single-stream in eager mode

One note for unified-memory platforms (GB10 and similar): a warm page cache (for example right after downloading the checkpoint) can starve the GPU allocator partway through model load. Dropping caches (`sync; echo 3 > /proc/sys/vm/drop_caches`) before launch avoids it.

Thanks for the PR - this arch runs nicely on Spark-class hardware.

**jhsmith409** (2026-08-26T18:42:20Z):

Another SM120 data point, but a **different failure from the two above** — this one is **TP=1**, and it gets _past_ weight loading.

Setup: single **RTX PRO 6000 Blackwell Max-Q (SM120, 96GB)**, driver 610.57.04, image `vllm/vllm-openai:qwen38-flash-next` (`sha256:fc120ece0a38`, vllm `0.1.dev20073+g8e685d198`, torch 2.13.0+cu130), checkpoint `RadixArk/Qwen3.8-Flash-Next-NVFP4`, `VLLM_PLE_CPU_OFFLOAD=1`, `--tensor-parallel-size 1 --gpu-memory-utilization 0.93 --max-model-len 65536 --no-enable-flashinfer-autotune`.

Everything up to warmup succeeds: 206 shards load in \~2 min, torch.compile 14.7s, CUDA graph capture 6s / 0.24 GiB, KV cache 11.55 GiB (440,027 tokens). No OOM, no SM120 kernel problem. Startup then **hangs silently and indefinitely** — no exception, no further log lines, EngineCore pinned at 100% CPU, nothing binds the port. Host RAM never rises, so the \~48 GB PLE table is never loaded.

Notably `VLLM_PLE_OFFLOAD_READY_TIMEOUT` does **not** bound this — I set it to 120s and the process was still spinning 6 minutes later. Neither hang reaches `wait_ple_offload_ready`.

**Hang 1 — with CUDA graphs (deadlock between the captured graph and the offload thread):**



```
Thread (active, 100% CPU): "MainThread"
    replay (torch/cuda/graphs.py:186)
    __call__ (vllm/compilation/cuda_graph.py:360)
    forward (vllm/models/qwen3_8_flash_next/nvidia/model.py:464)
    run_pw_graph (vllm/v1/worker/gpu/cudagraph_utils.py:436)
    execute_model (vllm/v1/worker/gpu/model_runner.py:1721)
    warmup_kernels (vllm/v1/worker/gpu/warmup.py:330)
    compile_or_warm_up_model (vllm/v1/worker/gpu_worker.py:925)
    collective_rpc (vllm/v1/executor/uniproc_executor.py:103)
    _initialize_kv_caches (vllm/v1/engine/core.py:331)

Thread (idle): "ple-offload-dp0"
    wait (threading.py:355)
    get (queue.py:171)
    _request_loop (vllm/v1/ple_offload/connector.py:263)
```

The captured graph appears to block on the PLE `cuStreamWaitValue32(flag==1)` while `ple-offload-dp0` blocks in `queue.get()` for a request that never arrives, so `replay()` never returns. Note `uniproc_executor` — the TP=1 path.

**Hang 2 — same run with `--enforce-eager` (graph deadlock gone, new hang behind it):**



```
Thread (active): "MainThread"
    _short_conv_dilated_prefill_batched (vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py:822)
    _short_conv_dilated_dispatch (vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py:1113)
    _short_conv (vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py:1189)
    qwen3_8_flash_next_ple_short_conv (vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py:1243)
    forward (vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py:1232)
    forward (vllm/models/qwen3_8_flash_next/nvidia/model.py:298)
    execute_model (vllm/v1/worker/gpu/model_runner.py:1726)
```

Identical across three py-spy samples 8s apart — no progress. So `--enforce-eager` breaks the graph/offload deadlock but the dilated short-conv prefill path then hangs on its own.

Both reproduce on every boot (5/5). Since @nichdiekuh and @OsakaTX both ran TP=2 and hit the loud `ngram_embedding.weight_scale` `ValueError` during loading, TP=1 + PLE offload may simply be an unexercised combination rather than the same bug.

Separately, on the hybrid-quant gate @OsakaTX identified: I applied the same workaround locally (env-gated, returning `Qwen3_8FlashNextPLEFp8EmbeddingMethod()` from `_get_ple_embedding_quant_method()` before the `isinstance(quant_config, Fp8Config)` check) and it does prevent the `weight_scale` error — but it does not change either hang above, which happen after loading. FWIW in this image the module is `vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py`, not `qwen4_exp`.

Happy to test patches on this hardware, or supply full boot logs / additional py-spy dumps.

**vadiklyutiy** (2026-08-27T00:13:44Z):

@peakcrosser7 as I see from comment above and from vllm receipt many configuration doesn’t work.

Could you please share what configs should work or were tested and what config shouldn’t work or failed during testing. (In my opinion better to add it to PR description)

**gau-nernst** (2026-08-27T01:36:57Z):

We have validated `Qwen/Qwen3.8-Flash-Next`, `Qwen/Qwen3.8-Flash-Next-FP8`, and `Inferact/Qwen3.8-Flash-Next-NVFP4` in e2e setting with GB300/GB200, H200, and MI355X, as shown in <https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next>. We will expand support for other configurations in future PRs.

**ZJY0516** (2026-08-27T02:18:26Z):

> @peakcrosser7 as I see from comment above and from vllm receipt many configuration doesn’t work.
>
> Could you please share what configs should work or were tested and what config shouldn’t work or failed during testing. (In my opinion better to add it to PR description)

on GB200, TP4, offload + TP4/TP2/DEP4, these are what I verified. @vadiklyutiy

**ZJY0516** (2026-08-27T02:40:29Z):

/ci run

**github-actions** (2026-08-27T02:40:47Z):

✅ Triggered [Buildkite CI #85734](https://buildkite.com/vllm/ci/builds/85734) for commit `ea5b4577ece1`.

**zzw09773** (2026-08-27T02:45:58Z):

While bringing this up on gfx942 we hit an out-of-bounds read that traces back to `CircularBufferSpec`
being classified as a normal token-to-slot cache group. Filed the kernel-side guard separately as
[vllm-project/vllm#53982](https://github.com/vllm-project/vllm/issues/53982); this is the model-side half.

`CircularBufferSpec.max_num_blocks_per_req()` returns `1`, and `QSAMetadataBuilder` computes the
ring’s slot mapping itself (`_build_circular_metadata` → `circular_qsa_slot_mapping`, which uses
only `block_table[req, 0]`). But `gpu_model_runner.py:7387-7390` maps `KVCacheSpecKind.MAMBA` to
`SlotMappingMode.NONE` and everything else to `TOKEN_TO_KV_SLOT`, and `CircularBufferSpec` is not a
`MambaSpec` subclass — it falls through `get_kv_cache_spec_kind()` to `UNKNOWN`. So the generic
per-group kernel runs on this group every step, indexing a 128-token-wide row with raw token
positions, and its result is then discarded.

Suggestion: let the spec declare its own mode (e.g. a `slot_mapping_mode` property on `KVCacheSpec`,
default `TOKEN_TO_KV_SLOT`, overridden to `NONE` on `MambaSpec` and on self-managing side caches)
rather than enumerating exceptions in the runner. That also removes one wasted kernel launch per
group per step.

One caveat if you take this: `NONE` makes `get_block_table_width` use `token_alignment=None`, which
narrows the row to 1. That is correct for `circular_qsa_slot_mapping` (column 0 only), but worth
double-checking nothing else depends on the 128-token-aligned width.

**zzw09773** (2026-08-27T02:47:29Z):

Two things, both aimed at the “which configurations work” table @vadiklyutiy asked for.

## 1. The AMD `ple_layer` cannot load the FP8 checkpoint’s n-gram table

With `Qwen/Qwen3.8-Flash-Next-FP8`, startup fails because `ngram_embedding.weight_scale` has no
corresponding parameter. In that checkpoint the PLE n-gram table is stored as `F8_E4M3` across 128
shards with a per-tensor scale (0.000199 in our copy); the AMD embedding path only accepts a BF16
table.

The two files are not symmetric on this. In this PR, `nvidia/ple_layer.py` (1170 lines) defines an
FP8 embedding method — `create_fp8_weight_parameter`, `create_fp8_scale_parameter`, a
`QuantizeMethodBase` subclass, and a `_dequantize_embeddings` helper — while `amd/ple_layer.py`
(1059 lines) contains **none** of those symbols.

Two ways to close it — we are running the first and have the second prepared:

- **Dequantize at load** — read the FP8 shards, multiply by the scale, store BF16. Simple, costs
  the memory difference and some load time. This is what we are serving with.
- **Keep it FP8 and dequantize at lookup** — port the NVIDIA method verbatim, i.e. register the
  FP8 weight + scale parameters and dequantize the gathered rows in `_dequantize_embeddings`.
  We have this prepared as a 7-edit patch against a clean `amd/ple_layer.py`; `F.embedding` on an
  `fp8_e4m3` tensor works on ROCm, so nothing else needs to change. Happy to open it as a PR if
  that is the direction you’d prefer.

## 2. gfx942 data point for the compatibility table

You listed GB300/GB200, H200 and MI355X. Here is **gfx942 / MI325X**, which is a different ROCm
target (CDNA3, FP8 is FNUZ, and the published ROCm image’s aiter prebuilts are gfx950-oriented):

|             |                                                                                                           |
| ----------- | --------------------------------------------------------------------------------------------------------- |
| Hardware    | 8× MI325X (gfx942), ROCm 7.2.3                                                                            |
| Parallelism | **TP8 with `--enable-expert-parallel`**                                                                   |
| Context     | `--max-model-len 262144`                                                                                  |
| Speculative | MTP, `num_speculative_tokens=3`, acceptance length 3.32                                                   |
| Throughput  | 152 tok/s single stream; 676 tok/s aggregate at 8-way concurrency                                         |
| Correctness | needle-in-haystack to 250k 3/3; zh-TW output, `reasoning_content`, `qwen3_coder` tool parsing all correct |

One constraint worth putting in the table because it is arithmetic, not a bug:
**pure tensor parallelism is not valid for this checkpoint.** `moe_intermediate_size = 640 = 5×128`
and the FP8 `weight_block_size` is `[128, 128]`, so TP2/4/8 give 320/160/80 per rank, none of which
is a multiple of 128. Expert parallelism is required. (The shared expert is in
`modules_to_not_convert`, i.e. BF16, so it is not subject to this.)

Getting there needed three ROCm-side fixes beyond the PLE one — the spec-decode metadata allowlist,
an aiter JIT race, and a slot-mapping out-of-bounds read ([#53982](https://github.com/aarnphm/aarnphm.github.io/issues/53982); the model-side half of that one
is in a separate comment on this PR). Filed separately so they don’t clutter this thread.

### Reproduction

Serve `Qwen/Qwen3.8-Flash-Next-FP8` on an AMD GPU (so the `amd/` module tree is selected).
Startup fails on `ngram_embedding.weight_scale` having no corresponding parameter. (We have not
run the `nvidia/` tree; it is the one that defines the FP8 embedding method.)

### What we verified / did not verify

- **Verified:** the symbol asymmetry between the two `ple_layer.py` files in this PR; the load
  failure and that both remedies clear it; all numbers in the table.
- **Not** verified: the native-FP8 port under long-running production load — we validated it
  structurally (anchors, dry-run line count, idempotency, AST) but are currently serving the
  dequantize-at-load variant.
- **Not** verified: any non-FP8 checkpoint on gfx942.

**peakcrosser7** (2026-08-27T03:32:24Z):

@jhsmith409 Thanks for your feedback. We will add PLE-offload support for NVFP4 quantization as soon as possible. Once available, you can retry using the NVFP4 checkpoint provided by Inferact. We will also investigate your reported issue and work on a fix.

**peakcrosser7** (2026-08-27T03:39:22Z):

@vadiklyutiy Thank you for the review. We have updated the PR description with the configurations we tested and validated. We will also address your code suggestions shortly.

**LucasWilkinson** (2026-08-27T04:47:26Z):

Following up on [vllm-project/vllm#53896 (comment)](https://github.com/vllm-project/vllm/pull/53896#discussion_r3868334388) made a proposal here: [peakcrosser7/vllm#6](https://github.com/peakcrosser7/vllm/pull/6) on how we can bring this closer inline with current main. I havent had a change to iterate on `_get_packed_kv_cache_groups` (vibecoded function for now; going to bed) as much as i would like (heading to bed) but directionally I think this right.

**peakcrosser7** (2026-08-27T05:39:15Z):

@LucasWilkinson Thanks for the review and the PR. I’ll refer to your changes and consider how best to refine the implementation.

**ZJY0516** (2026-08-27T06:34:59Z):

/ci run

**github-actions** (2026-08-27T06:35:54Z):

✅ Triggered [Buildkite CI #85763](https://buildkite.com/vllm/ci/builds/85763) for commit `d9f64bd7094d`.

**peakcrosser7** (2026-08-27T06:42:42Z):

/ci run

**github-actions** (2026-08-27T06:43:01Z):

✅ Triggered [Buildkite CI #85764](https://buildkite.com/vllm/ci/builds/85764) for commit `d69520dd4559`.

**zzw09773** (2026-08-27T07:31:06Z):

On the ROCm path (`models/qwen3_8_flash_next/amd/qsa.py`), `Qwen3_8FlashNextQSAFlashAttentionBackend`
subclasses `FlashAttentionBackend` and does not override `supports_batch_invariance()`, so it
inherits `return True` from `vllm/v1/attention/backends/flash_attn.py`. But the QSA implementation
does not use FlashAttention: `Qwen3_8FlashNextQSAFlashAttentionImpl.forward_qsa` calls the model’s
own `qsa_sparse_paged_attention` Triton kernel.

That kernel is not batch-invariant, and the reason is concrete — the split-K reduction depth is a
function of the number of query rows in the batch:



```python
base_programs = (
  q.shape[0] * k_cache.shape[2]
)  # q.shape[0] = query rows in this batch
small_profile_limit = 8 if block_m <= 8 else 4
if base_programs <= small_profile_limit:
  block_n, target_splits, partial_warps = 16, 64, 4
elif base_programs < 32:
  block_n, target_splits, partial_warps = 16, 32, 4
elif base_programs <= 256:
  block_n, target_splits, partial_warps = 64, 8, 2
elif base_programs <= 512:
  block_n, target_splits, partial_warps = 64, 4, 2
else:
  block_n, target_splits, partial_warps = 64, 1, 2
...
num_splits = min(max_useful_splits, target_splits)
```

`num_splits` sets how many partial accumulators `_qsa_merge_splitk_kernel` combines. Crossing any
of those thresholds changes the reduction tree, and therefore changes the floating-point rounding
of the same request’s attention output. That is precisely what `VLLM_BATCH_INVARIANT` is supposed
to rule out.

## Why this is currently latent, and why it will not stay that way

On this model the flag cannot be reached today: `vllm/v1/attention/selector.py` raises before the
attention backend is consulted, because the GDN/mamba backend does not declare batch invariance:



```python
if (
  envs.VLLM_BATCH_INVARIANT
  and not mamba_attn_backend.supports_batch_invariance()
):
  raise RuntimeError(
    f'VLLM batch_invariant mode is not supported for {mamba_attn_backend.get_name()}.'
  )
```

So the incorrect claim is masked by an unrelated guard. The moment mamba/GDN gains batch-invariance
support ([#48613](https://github.com/aarnphm/aarnphm.github.io/issues/48613) tracks exactly that), this model would start up with
`VLLM_BATCH_INVARIANT=1` and **silently** produce batch-dependent results — the failure mode the
flag exists to prevent. A backend-level capability claim that is only true by accident of a
neighbouring guard is worth fixing before it becomes load-bearing.

## Suggestions

1. **One line, immediate:** override `supports_batch_invariance()` to return `False` on
   `Qwen3_8FlashNextQSAFlashAttentionBackend`. `AttentionBackend` already defaults to `False`; the
   `True` is inherited only because the class reuses `FlashAttentionBackend` for its metadata
   builder.
2. **If QSA should support the flag:** make the profile selection independent of `q.shape[0]` when
   `vllm_is_batch_invariant()` is set — pin `(block_n, target_splits, partial_warps)` to one
   profile so `num_splits` no longer depends on batch composition.

Option 1 is the correct default: declaring `False` costs nothing but an error message, while
declaring `True` wrongly costs silent numerical divergence.

**peakcrosser7** (2026-08-27T08:08:42Z):

@zzw09773 Thank you very much for your review and suggestions. ROCm support in the current PR is not yet complete, and we will address the remaining gaps as soon as possible.

**vowstar** (2026-08-27T08:30:59Z):

Build break on sm_80, one hunk.

In csrc/libtorch_stable/ops.h, fused_gdn_decode_post_conv_mtp is declared under VLLM_ENABLE_FUSED_KDA_DECODE but used under VLLM_ENABLE_FUSED_GDN_DECODE. On sm_90+ both macros are set so nothing shows. On sm_80 only the GDN macro is set and the build fails with “not declared in this scope”. Giving the function its own VLLM_ENABLE_FUSED_GDN_DECODE guard fixes it.

Tested on 5x CMP 170HX (sm_80), pipeline parallel 5, with PR [#53899](https://github.com/aarnphm/aarnphm.github.io/issues/53899) at [`f561eca`](https://github.com/aarnphm/aarnphm.github.io/commit/f561eca6c). Three more small patches were needed for PP5, all in model code:

| where                                  | what                                                                                                   |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| v1/worker/gpu_worker.py                | enable the PLE offload client only on rank 0 and drop the PP=1 validation, only rank 0 owns PLE layers |
| models/qwen4_exp/nvidia/model.py       | skip hyper_connection_mixer weights on ranks where the mixer is None                                   |
| models/qwen4_exp/nvidia/model_state.py | let non-first PP ranks disable ngram state instead of raising                                          |

With those, plain PP5 serving works. MTP with PP does not: boot dies with a CUDA illegal memory access in the target model once speculative decoding makes decode query_len 3, on an intermediate pipeline rank, in both the V2 generic speculator and the V1 dedicated proposer paths. This is consistent with the recipe supporting only single-node TP or TEP, but worth knowing PP+MTP fails at runtime rather than at config time.

**peakcrosser7** (2026-08-27T08:36:59Z):

/ci run

**github-actions** (2026-08-27T08:37:18Z):

✅ Triggered [Buildkite CI #85787](https://buildkite.com/vllm/ci/builds/85787) for commit `2a4cd640ff1a`.

**peakcrosser7** (2026-08-27T11:01:47Z):

> While bringing this up on gfx942 we hit an out-of-bounds read that traces back to `CircularBufferSpec` being classified as a normal token-to-slot cache group. Filed the kernel-side guard separately as [#53982](https://github.com/aarnphm/aarnphm.github.io/issues/53982); this is the model-side half.
>
> `CircularBufferSpec.max_num_blocks_per_req()` returns `1`, and `QSAMetadataBuilder` computes the ring’s slot mapping itself (`_build_circular_metadata` → `circular_qsa_slot_mapping`, which uses only `block_table[req, 0]`). But `gpu_model_runner.py:7387-7390` maps `KVCacheSpecKind.MAMBA` to `SlotMappingMode.NONE` and everything else to `TOKEN_TO_KV_SLOT`, and `CircularBufferSpec` is not a `MambaSpec` subclass — it falls through `get_kv_cache_spec_kind()` to `UNKNOWN`. So the generic per-group kernel runs on this group every step, indexing a 128-token-wide row with raw token positions, and its result is then discarded.
>
> Suggestion: let the spec declare its own mode (e.g. a `slot_mapping_mode` property on `KVCacheSpec`, default `TOKEN_TO_KV_SLOT`, overridden to `NONE` on `MambaSpec` and on self-managing side caches) rather than enumerating exceptions in the runner. That also removes one wasted kernel launch per group per step.
>
> One caveat if you take this: `NONE` makes `get_block_table_width` use `token_alignment=None`, which narrows the row to 1. That is correct for `circular_qsa_slot_mapping` (column 0 only), but worth double-checking nothing else depends on the 128-token-aligned width.

The current `_get_slot_mapping_mode()` already checks `isinstance(kv_cache_spec, CircularBufferSpec)` and returns `SlotMappingMode.NONE` when that condition is met, so it should not incorrectly return `TOKEN_TO_KV_SLOT`.
That said, I think your suggestion is reasonable. We’ll discuss how best to adjust the design.

**zzw09773** (2026-08-27T11:11:59Z):

On 8x MI325X (gfx942, ROCm 7.2.3, TP8+EP, MTP with `num_speculative_tokens=3`) this PR’s PLE
n-gram context is wrong on **every** decode step in the **default** configuration. Across two
requests we could classify 154 decode steps and **none** of them received a correct n-gram
context. The cause is that
`_prepare_ngram_context` reads the CPU-side token mirror at a point where, under async scheduling,
that mirror still holds the optimistic `-1` placeholders that speculative decoding writes before
acceptance is known.

## The code

`vllm/v1/worker/gpu_model_runner.py` (added by this PR; there is no `_prepare_ngram_context` on
`main`):



```python
num_computed = self.input_batch.num_computed_tokens_cpu
token_ids = self.input_batch.token_ids_cpu
for req_idx in range(num_reqs):
  end = int(num_computed[req_idx])
  start = max(0, end - self.ngram_context_len)
  context_tokens = token_ids[req_idx, start:end]
  ...
  context_cpu[req_idx, -context_tokens.size :] = context_tokens
```

`end` is fine — in `mamba_cache_mode == "align"` the deferred speculative correction is forced
before `_preprocess` runs, so `num_computed_tokens_cpu` is already corrected. The problem is the
**contents** of `token_ids_cpu` at `[end - ngram_context_len : end]`.

Under async scheduling with speculative decoding, `CachedRequestState.output_token_ids` is extended
with optimistic placeholders:



```python
optimistic_num_accepted = req_state.prev_num_draft_len
req_state.output_token_ids.extend([-1] * optimistic_num_accepted)
```

and `GPUInputBatch` documents the same thing: _“async spec decode adds optimistic placeholders that
may exceed the actual acceptance count.”_ Those placeholders reach `token_ids_cpu` through
`_update_states`, and are only replaced later.

## What we measured

Instrumented `_prepare_ngram_context` to log, per call, the sliced context `ctx`, the surrounding
row `token_ids_cpu[req, end-4:end+2]`, and `output_token_ids[-3:]`. Three requests, 178 calls:

| phase         | `end` | `ctx` (the two ids fed to the n-gram)               | `output_token_ids[-3:]` |
| ------------- | ----- | --------------------------------------------------- | ----------------------- |
| prefill chunk | 24    | `[321, 264]` — real tokens                          | `[]`                    |
| prefill tail  | 36    | `[248069, 271]` — real tokens                       | `[]`                    |
| decode        | 39    | **`[-1, -1]`**                                      | `[-1, -1, -1]`          |
| decode        | 43    | `[198, 248045]` — stale, not this position’s tokens | `[-1, -1, -1]`          |
| decode        | 47    | **`[-1, -1]`**                                      | `[-1, -1, -1]`          |
| decode        | 63    | **`[-1, 0]`**                                       | `[-1, -1, -1]`          |
| decode        | 67    | **`[0, 0]`**                                        | `[-1, -1, -1]`          |

Only the two prefill chunks got real tokens. Reconstructing each request’s committed token
sequence from the per-step accepted tokens and comparing against it:

| request | classifiable decode steps | correct | `ctx` contains `-1` | `ctx` contains `0` | real but stale ids |
| ------- | ------------------------- | ------- | ------------------- | ------------------ | ------------------ |
| R       | 83                        | **0**   | 26                  | 54                 | 3                  |
| X       | 71                        | **0**   | 37                  | 33                 | 1                  |

The first decode step is already wrong: `end=39`, `ctx=[-1, -1]`, where the correct context is
`[271, 248069]`. The two requests use the identical prompt and produce _different_ sequences of
wrong values.

Two distinct wrong values appear, and they matter differently:

- **`-1`** is the optimistic placeholder described above — a sentinel that was never meant to reach
  an embedding lookup.
- **`0`** is a **valid token id**, not a sentinel. It comes from positions of `token_ids_cpu` that
  were never written for this request (the buffer is zero-initialised). So those steps do not fail
  or clamp; they silently feed the embedding for token id 0.

## Why this surfaced as nondeterminism

We found it while chasing run-to-run nondeterminism with greedy decoding. A per-layer activation
bisection put the first divergence at **layer 1’s PLE output**, with every other layer bit-identical
up to that step. Within the PLE input, only the **first two** of the four query rows differed —
which is exactly the set of rows whose 2-/3-grams reach outside the current step and therefore
consume `ngram_context` (`ngram_size = 3`). Running with `--no-async-scheduling` made 16/16 probe
runs bit-identical, including the first request after boot.

## The same PR already contains the correct pattern, twice

1. `preprocess_mamba` reads CPU-side state too, and this PR’s runner explicitly forces the deferred
   correction before it, with a comment saying why:



   ```python
   if self.cache_config.mamba_cache_mode == 'align':
     # preprocess_mamba reads req_state.num_computed_tokens (CPU)
     # to decide copy operations, so we must apply deferred
     # corrections before it runs.
     if deferred_state_corrections_fn:
       deferred_state_corrections_fn()
   ```

   `_prepare_ngram_context` reads CPU-side state as well but has no equivalent guard.

2. The PR also ships a model-runner-V2 implementation whose class docstring is _“Add rollback-safe
   PLE n-gram context to the model inputs.”_ — i.e. the author identified this exact hazard. But
   **that path is unreachable for this model**: `Qwen3_8FlashNextForCausalLM` /
   `Qwen3_8FlashNextForConditionalGeneration` are not in
   `DEFAULT_V2_MODEL_RUNNER_ARCHITECTURES`, and the model is hybrid, so
   `_is_default_v2_model_runner_model()` returns `False` and V1 always runs. (Checked against
   `main`: the architecture list there does not contain them either.)

So the safe design exists in the PR, on a path this model cannot take.

## Suggested fixes, cheapest first

1. **Filter placeholders.** Treat `-1` (and any not-yet-written slot) as “no context” and fall back
   to the eos fill the function already uses, rather than feeding it to the embedding.
2. **Take the confirmed tail.** Build the context from the last _confirmed_ entries of
   `req_state.output_token_ids` rather than from the optimistic CPU mirror.
3. **Order it like mamba.** Call the n-gram context preparation after the deferred correction and
   after the sampled-token write-back, the way `preprocess_mamba` is handled today.
4. **Or make the V2 “rollback-safe” path reachable** for these architectures.

(1) alone removes the `-1` case; it does **not** fix the stale-value case, so we would expect (2)
or (3) to be the real fix.

## Reproduction

Serve `Qwen/Qwen3.8-Flash-Next-FP8` with `--speculative-config '{"method":"mtp",
"num_speculative_tokens":3}'` and default scheduling. The most direct check is to log `ctx` in
`_prepare_ngram_context` for a single greedy request: it shows `-1` / `0` from the first decode step.
Behaviourally: send a prompt with `temperature=0`, run a burst of 8 concurrent other requests, then
send the same prompt again — the two greedy generations diverge (in our probes almost every such
pair did; two back-to-back sends without the burst often stay identical, so the burst matters).
Adding `--no-async-scheduling` makes every repeat bit-identical.

**zzw09773** (2026-08-27T11:13:48Z):

You’re right, and thanks for checking. Our observation came from an older snapshot of this branch (the `b0731de1c` build we run on gfx942), where the runner only special-cased `KVCacheSpecKind.MAMBA` inline. The current head (`2a4cd640`) has `_get_slot_mapping_mode()` returning `SlotMappingMode.NONE` for `CircularBufferSpec`, which resolves the model-side half of what we hit — so please treat that part of our note as already addressed.

Two things still stand independently of it:

- the kernel-side bound in `ComputeSlotMappingKernel` ([#53982](https://github.com/aarnphm/aarnphm.github.io/issues/53982)) is defence in depth for any other group whose block table is narrower than the sequence, and is being picked up there;
- the design suggestion (let the spec declare its mode rather than the runner enumerating types) is just a preference — the explicit `isinstance` check is fine for this PR.

Apologies for the stale reference; we’ll re-check against the PR head before commenting further.

**zzw09773** (2026-08-27T11:16:16Z):

One refinement to the fix list above, from re-reading the PR head (`2a4cd640`):

`_prepare_ngram_context` already contains an `is_token_ids`-based sanitiser, but it is gated on `enable_prompt_embeds`:



```python
if self.enable_prompt_embeds and not is_token_ids[req_idx, start:end].all():
  safe_tokens = context_tokens.copy()
  safe_tokens[~is_token_ids[req_idx, start:end]] = eos_token_id
  context_tokens = safe_tokens
```

Ungating it (one line) would cover the never-written case — the `0` values, which were 54 of the 83 wrong steps in our trace — because those positions have `is_token_ids == False`.

It would **not** cover the `-1` case: in `_update_states` the placeholder is written into `token_ids_cpu` and `is_token_ids[...] = True` is set right after it (head `:1598-1604`), so the flag does not distinguish a placeholder from a real token. That is why we still think the real fix is to build the context from confirmed tokens or to order the call after the write-back, as in the note above; the ungated sanitiser is a cheap first step, not a complete one.

**ZJY0516** (2026-08-27T11:27:18Z):

/ci run nightly

**github-actions** (2026-08-27T11:27:36Z):

✅ Triggered [Buildkite CI #85802](https://buildkite.com/vllm/ci/builds/85802) for commit `2a4cd640ff1a`.

**dolf3131** (2026-08-27T12:37:15Z):

A data point for the “which configurations work” table, on **prefix caching** — it does work on this model, but only with `--mamba-cache-mode align`, and the 0%-hit-rate reports circulating are explained by two things rather than by the feature being broken.

Hardware: single DGX Spark (GB10, sm_121a, 121 GiB unified), `Inferact/Qwen3.8-Flash-Next-NVFP4`, `VLLM_PLE_CPU_OFFLOAD=1` with the table paged to swap, TP=1, MTP k=2, image `vllm/vllm-openai:qwen38-flash-next-arm64-cu130` (`0.1.dev20073+g8e685d198`).

**1. `--enable-prefix-caching` alone is inert.** `mamba_cache_mode` defaults to `"none"`, and nothing promotes it to `"align"` when prefix caching is switched on — the docstring in `config/cache.py` says align “is the default when prefix caching is enabled”, but `arg_utils.py` just forwards whatever the user passed, and `config/vllm.py` only ever tells the user to `pass --mamba-cache-mode align`. Measured with prefix caching on and the mode left at its default: **0 hits in 813 queries.**

**2. With `align`, the attention block size is forced to 1600 tokens** so it matches the mamba page size (`Setting attention block size to 1600 tokens ...`, `Padding mamba page size by 0.88% ...`). Any prompt shorter than 1600 tokens therefore fills no block and can never produce a hit. A benchmark built from short prompts will report 0% no matter how repetitive it is — worth knowing before concluding the feature is dead.

**With both flags and a prompt long enough to fill blocks**, an 8,053-token prompt sent three times:

| request | TTFT       | prefill         | prefix-cache hits           |
| ------- | ---------- | --------------- | --------------------------- |
| 1       | 5.41 s     | 1,497 tok/s     | 0                           |
| 2       | 3.81 s     | 2,128 tok/s     | 0                           |
| 3       | **1.01 s** | **8,015 tok/s** | **6,400 tokens** (4 blocks) |

**5.4x on TTFT** once the blocks commit. Hits only appeared on the third identical request, not the second, which I assume is align mode only checkpointing state at block boundaries as the docstring describes.

**No decode regression:** 28.19 tok/s without prefix caching versus 28.25 with it, same four prompts, same boot conditions — those prompts are 87 tokens so they cannot hit, which is exactly the point: turning it on costs nothing when it cannot help.

**No CUBLAS failure.** There is a report circulating that `--enable-prefix-caching` must be avoided on GB10 because a GDN `in_proj` GEMM hits `CUBLAS_STATUS_INTERNAL_ERROR` on the cached-block path with a second identical prompt. I could not reproduce that here: repeated identical prompts, short and long, complete normally with `align` set. I cannot rule out that the report was made with `mamba_cache_mode` at its default — which is the half-enabled configuration in point 1, where attention blocks are reused while the GDN state is not cacheable. If someone hitting that crash can confirm which mode they ran, it would be worth knowing whether the guard belongs in the config validator.

Suggestion, cheap either way: raise an error, or at least a warning, when `enable_prefix_caching` is set while `mamba_cache_mode == "none"` on a hybrid model. As it stands the flag is silently a no-op, and possibly worse than a no-op.

**zzw09773** (2026-08-27T13:37:21Z):

Two ROCm data points for the same table (8x MI325X, TP8+EP, MTP k=3, this branch at `b0731de1c`):

1. **The align promotion does fire here.** With only `--enable-prefix-caching`, startup logs `Mamba cache mode is set to 'align' for Qwen3_8FlashNextForConditionalGeneration by default when prefix caching is enabled`. That comes from `vllm/model_executor/models/config.py` (the hybrid-model `verify_and_update_config` hook, lines \~613-616 on both the PR head `2a4cd640` and `main @ fd57c4b7`). So on this stack the flag is not inert — if it stays at `"none"` on your build, it may be that the hybrid config hook is not reached for your model path rather than the promotion being absent; worth a startup-log check for that exact line.

2. **A second aligned-block-size number, not an explanation.** We measure **400** on TP=8 / gfx942 / FP8 (printed `cache_config.block_size` at the point the QSA ring spec asserts on it) against your 1600 on TP=1 / GB10 / NVFP4. We have _not_ derived where the difference comes from: both the mamba page and the per-token attention page shrink with TP, so the ratio should not change with TP alone, and `kernel_block_alignment_size` in the platform hook is backend-dependent — so please read these as two data points rather than a TP scaling rule. Note `/metrics cache_config_info` still reports the pre-hook values (`block_size=8`, `mamba_block_size=16`) — it snapshots the config before the platform hook rewrites it, so it is not a reliable way to read the runtime block size.

Consistent with your observation on hits: our 31.5k-token prompts do hit once their blocks are committed — repeat prefills went from 20–250 s TTFT (cold, 8–64 concurrent) to 0.8–3.9 s.

**SirTificate** (2026-08-27T15:30:29Z):

For anyone landing here with workstation Blackwell: we got this model serving properly on **2× RTX PRO 6000 (SM120), TP=2** — both the FP8 and the NVFP4 checkpoint, with MTP and the full 262k context — and wrote it up as a recipe:

**<https://github.com/SirTificate/qwen38-flash-next-2x-rtx6000-sm120>**

Highlights relevant to this thread:

- **NVFP4 + PLE offload works** with @OsakaTX’s hybrid-quant gate workaround (we made it env-conditional, `VLLM_PLE_FORCE_FP8=1` — 3-line patch in the repo). NVFP4 ends up ahead of FP8: 40.5 vs 63.7 GiB weights/rank, 3.1M vs 1.85M KV tokens, 161 vs 149 tok/s single-stream.
- **`--disable-custom-all-reduce` is required on a _homogeneous_ pair too** — we hit the same `custom_all_reduce.cuh:164 'invalid argument'` graph-capture crash net-snix reported on Max-Q+Workstation, on two identical Workstation cards.
- MTP acceptance is strongly workload-dependent: 44 % on mixed chat, **87–90 %** (mean accept len 3.6–3.7 of 4) on math/instruction benchmarks. \~1,065 tok/s aggregate at 32 concurrent, still climbing at the seq cap.
- Quality sanity check on the NVFP4 quant: GSM8K 98.0, HumanEval+ 95.1, MATH-500 92.4, IFEval 84.3.

Thanks for the day-0 support — the PLE offload machinery in the image is what makes this class of hardware viable for the 51B n-gram table.

**zzw09773** (2026-08-27T15:47:59Z):

On 8× MI325X (gfx942, ROCm 7.2.3, TP8+EP) the server fails to start for
`num_speculative_tokens = 5` — which is the value the published recipe suggests. The failure is an
`AssertionError` during KV-cache spec construction, and the container exits unhealthy with no
actionable message.

## Where

`vllm/models/qwen3_8_flash_next/common/qsa_cache.py`, `QSAKeyStateCache.get_kv_cache_spec`:



```python
# ... rounded up to whole groups so
# the ring divides the attention block size (it joins the LCM that sets
# the scheduler block size). Anything narrower lets a rejected draft row
# overwrite a committed key the next step needs to close the group.
span = self.compress_ratio + vllm_config.num_speculative_tokens
capacity = self.compress_ratio * cdiv(span, self.compress_ratio)
assert self.cache_config.block_size % capacity == 0, (
  f'QSA ring capacity {capacity} must divide the attention block '
  f'size {self.cache_config.block_size}'
)
```

The comment states that the ring capacity “joins the LCM that sets the scheduler block size” — i.e.
the divisibility is supposed to hold by construction. **Empirically it does not.**

## What we measure

We printed `cache_config.block_size` immediately before the assert, on every worker:



```
cache_config.block_size=16  mamba_block_size=16  capacity=8  span=7  compress_ratio=4
```

With `compress_ratio = 4` (`indexer_compress_ratio` in the checkpoint config) and B = 16:

| `num_speculative_tokens` | `span` | `capacity` | `16 % capacity` | result                      |
| ------------------------ | ------ | ---------- | --------------- | --------------------------- |
| 0                        | 4      | 4          | 0               | starts                      |
| 1–4                      | 5–8    | 8          | 0               | starts                      |
| **5–8**                  | 9–12   | **12**     | **4**           | **assert, exits unhealthy** |
| 9–12                     | 13–16  | 16         | 0               | starts                      |
| 13–16                    | 17–20  | 20         | 16              | assert                      |

So the usable set on this configuration is **non-contiguous**: `{0..4}` and `{9..12}` work,
`{5..8}` and `{13..16}` do not. The published recipe’s `nst = 5` falls in the first hole. We
confirmed 5 fails twice; 1, 3 and 0 all start normally.

### A note for anyone debugging this assert

`block_size` takes three different values at three points in startup, and only the middle one is
the one this assert compares against:

| where                                                         | value  | why                                                    |
| ------------------------------------------------------------- | ------ | ------------------------------------------------------ |
| `/metrics` `vllm:cache_config_info`                           | 8      | the config snapshot published before the platform hook |
| **this assert**, in `get_kv_cache_spec`                       | **16** | after `Platform.check_and_update_config` adjusted it   |
| the per-group spec seen by the mamba state kernels at runtime | 400    | after KV-cache-group page unification                  |

An operator who reads the assert message (“must divide the attention block size 16”) and then
checks `/metrics` will see 8 and conclude the message is wrong. Including the effective value’s
provenance in the message, or asserting against a value the operator can observe, would help.

## Why we think the invariant is not established

`capacity` is a function of `(compress_ratio, num_speculative_tokens)`. `block_size`, on this
model, is resolved from the hybrid-allocator page-size relationship between the GDN/mamba state and
the attention KV — quantities with no semantic relationship to the ring. The divisibility is
therefore coincidental, and it holds only while `cdiv(span, compress_ratio)` happens to be a power
of two (i.e. `nst ≤ 4` here).

We did not trace where the “LCM” the comment refers to is computed, so we cannot say whether the
ring spec is simply absent from it or whether it runs before `num_speculative_tokens` is known.
That part is inference from the observed numbers, not from reading that code path.

## Suggestions, cheapest first

1. **Make the error actionable.** Replace the assert with a raised error that names the supported
   values, e.g. _“QSA ring capacity 12 does not divide attention block size 8; supported
   num_speculative_tokens for this configuration: 0–4”_. Today the operator sees an
   `AssertionError` and an unhealthy container, with nothing indicating that the fix is to change
   `num_speculative_tokens`.
2. **Round the capacity to a power of two** (`capacity = 1 << (span - 1).bit_length()` or similar).
   That preserves the ”≥ span” property the comment requires, and it divides any power-of-two
   block size. The cost is at most 2× the ring, which is one block per request.
3. **Or genuinely include the ring capacity in whatever computes `block_size`**, so the comment’s
   claim becomes true.

### Reproduction

Serve `Qwen/Qwen3.8-Flash-Next-FP8` on ROCm with
`--speculative-config '{"method":"mtp","num_speculative_tokens":5}'`. The engine raises during KV
cache spec construction. `nst=3` on the same command line starts and serves normally.

### What we verified / did not verify

- **Verified by measurement:** `cache_config.block_size = 16` printed immediately before the
  assert, identical on all workers; that `nst ∈ {0,1,3}` start and `nst = 5` does not (twice).
- **Verified by arithmetic:** the capacity table above.
- **Observed but not derived:** the three-value table. We measured 8 in `/metrics`, 16 at the
  assert, and 400 in the per-group spec at the mamba kernel call sites. We did **not** derive why
  the last one is 400; we are reporting where each number comes from, not explaining the
  page-unification arithmetic.
- **Not** verified: where the “LCM that sets the scheduler block size” is computed, and therefore
  whether the ring is missing from it or the ordering is wrong. Stated as inference above.
- **Not** verified: whether any other configuration (different `compress_ratio`, different
  hardware, a `block_size` divisible by 3) makes `nst ∈ {5..8}` work. Our claim is scoped to this
  configuration. Note that `nst ∈ {9..12}` passes the assert here but we did **not** run it — we
  only checked the divisibility, not that the model works at that lookahead.

**cerdicc** (2026-08-27T19:19:16Z):

## SM80 (Ampere) enablement: working e5m2 route, verified on 4× CMP 170HX

Since the sm_80 build issue was raised (thanks @vowstar), sharing a **working Ampere route** that we’ve deployed and measured on real hardware — hoping it helps shape SM80 support in this PR.

### Approach (no kernel changes, \~20 lines)

SM80 has no `fp8e4nv` in Triton (needs SM89+), but **`fp8_e5m2` compiles fine** (software-emulated → BF16 compute on native tensor cores). We re-routed the model’s FP8 pipeline to e5m2 on top of the official `qwen38-flash-next` docker image:

1. `platforms/interface.py` → `fp8_dtype()` returns `float8_e5m2` when `capability == (8, 0)`
2. `quantization/fp8.py` → `create_weights`: hardcoded `e4m3fn` replaced with `current_platform.fp8_dtype()`
3. `quantization/utils/fp8_utils.py` → non-e4m3 dtypes route through the in-function Triton fallback (the prebuilt CUDA `per_token_group_quant_8bit_fp8` op rejects e5m2)
4. `fused_moe/experts/triton_moe.py` → admit `float8_e5m2` in the dtype allowlist + both `compute_type` tables (→ `tl.bfloat16`)

The checkpoint stays official e4m3 on disk; loader-side `copy_()` converts to e5m2 at load time (same 1 byte/weight, block scales preserved). Same spirit as the MiniMax-M3 SM80 work (Hakureirm/minimax-m3-ampere).

### Measured on 4× CMP 170HX 64GB (GA100, PCIe Gen2 x16, no NVLink), TP4

| Metric                     | Result                                            |
| -------------------------- | ------------------------------------------------- |
| max-model-len              | **262,144** (native max)                          |
| piecewise CUDA graphs      | 51/51 captured                                    |
| decode (stream, greedy)    | **37.6–37.9 tok/s** sustained; 52.4 peak @32K ctx |
| TTFT (warm)                | 262 ms; 64K prefill ≈ 56,000 tok/s                |
| tool calls (`qwen3_coder`) | round-trips correctly                             |

### Two findings worth noting for this PR

1. `--kv-cache-dtype fp8_e5m2` is hard-rejected by QSA (`qsa.py:187` — “requires a BF16 main KV cache”). Fine in practice since the MLA-compressed KV is small, but a friendlier error message might save others some debugging time.
2. MTP (`num_speculative_tokens=3`) measured **net-negative** on SM80: accept rates 65/42/28%, but per-step verification overhead on emulated-FP8 MoE eats the gains (21–22 tok/s vs 37.9 without). Possibly SM80-specific; fine elsewhere.

Full patch set + benchmarks: <https://github.com/cerdicc/qwen38-flashnext-sm80>

---

## 中文说明（Chinese summary）

自 @vowstar 提出 sm_80 构建问题以来，分享一条**已在真实硬件部署并实测的 Ampere 可行路线**，希望对本 PR 的 SM80 支持设计有帮助。

**方案**（无需修改内核，约 20 行）：SM80 的 Triton 不支持 `fp8e4nv`（需 SM89+），但 `fp8_e5m2` 可正常编译（软件模拟 → BF16 原生张量核心计算）。在官方 `qwen38-flash-next` 镜像之上将 FP8 管线整体改道 e5m2：平台层 dtype 开关、MoE 权重创建去硬编码、激活量化走 Triton 回退、MoE 白名单与计算类型表准入 e5m2（共四处五针）。权重保持官方 e4m3 原包，装载期数值转换，显存占用不变，块缩放保留。与 MiniMax-M3 的 SM80 工作（Hakureirm/minimax-m3-ampere）思路同源。

**实测**（4× CMP 170HX 64GB，GA100，PCIe Gen2 x16，无 NVLink，TP4）：上下文 262,144（原生上限）；分段 CUDA Graph 51/51 全部捕获；流式解码持续 37.6–37.9 tok/s（32K 档峰值 52.4）；热态 TTFT 262ms，64K prefill 约 56,000 tok/s；`qwen3_coder` 工具调用往返解析正确。

**两个值得注意的发现**：① QSA 硬性拒绝 fp8 KV（`qsa.py:187` 要求 BF16 主缓存），实际无碍（MLA 压缩 KV 很小），但建议加友好报错；② MTP 投机解码（×3）在 SM80 实测净负收益：接受率 65/42/28%，软模拟 FP8 MoE 的验证开销吃掉全部收益（21–22 tok/s，反低于关闭时的 37.9），可能为 SM80 特有。

完整补丁与基准：<https://github.com/cerdicc/qwen38-flashnext-sm80>

**jschmied** (2026-08-27T19:29:01Z):

**GB10 / TP=1 data point for the NVFP4 PLE gate, plus MTP numbers on the same box.**

Confirming @OsakaTX’s diagnosis (<ref slug="tags/issuecomment-5427847488">) on hardware this thread has not covered
— every existing report of it is TP=2 or multi-GPU. Single GB10 (DGX Spark, sm_121), TP=1,
`RadixArk/Qwen3.8-Flash-Next-NVFP4`, `VLLM_PLE_CPU_OFFLOAD=1`, image `g8e685d198`.

Same failure, same function:



```
ValueError: There is no module or parameter named 'ngram_embedding.weight_scale'
            in Qwen3_8FlashNextNGramEmbedding
```

Accepting `modelopt` / `modelopt_fp4` in `_get_ple_embedding_quant_method()` fixes it, and the
model then serves correctly — verified on content, not just on absence of errors:



```
"The capital of France is"  -> Paris.
"17 sheep, all but 9 die"   -> 9
iterative Fibonacci         -> correct
Rayleigh scattering (DE)    -> correct
```

**One caveat that cost us a while, worth folding into whatever lands.** Under PLE **CPU offload**
the GPU-side process must _not_ register `weight` / `weight_scale`. `load_weights()` retains only
`_offload_weight_scale`; a registered-but-never-loaded `weight_scale` then shadows it in
`_get_embedding_weight_scale()`, and the lookup dequantizes against an uninitialised value. That
fails **silently** — no error, correct-magnitude activations, fluent but wrong output. So the gate
needs to stay closed in the GPU process when offload is active:



```python
if envs.VLLM_PLE_CPU_OFFLOAD and not is_offload_process():
  return None
```

Resident cost on this box: **76.61 GiB** weights + 30.99 GiB KV, of 121.63 GiB, with a 64 GiB
swapfile carrying the cold part of the FP8 PLE table. So NVFP4 + PLE offload at TP=1 fits on one
Spark with headroom — noting the PR’s validation matrix currently lists PLE offload as BF16/FP8
only.

**MTP works on this configuration**, which may be useful for the recipe page since the numbers
differ from the single-stream-only ones circulating:

| concurrency | no spec |  MTP k=2 |  MTP k=3 |
| ----------: | ------: | -------: | -------: |
|           1 |    17.1 | **28.5** |     27.4 |
|           4 |    44.1 |     50.7 | **60.6** |
|           8 |    87.5 |     89.0 | **93.4** |

Acceptance k=2: 77.1% / 58.5% at positions 0-1, mean accepted length 2.36. k=3: 70.3% / 48.2% /
33.0%, mean 2.52.

**The optimal `num_speculative_tokens` shifts with load** — k=2 wins single-stream, k=3 wins from
c=4 up. k=3 drafts longer but costs more per step, and batching absorbs that cost. Since the
recipe page recommends k=3 and several reports recommend k=2, both are right for the concurrency
each was measured at; it may be worth stating the load assumption alongside the value.

Separately: `--max-num-seqs` matters far more than either. At c=8 the difference between the
value we first shipped (2) and 16 is **4x aggregate throughput**, and a request cap is
indistinguishable from saturation unless you also record
`vllm:request_queue_time_seconds_sum`. We nearly published a wrong ceiling because of it.

One more from this box, filed separately as [#54097](https://github.com/aarnphm/aarnphm.github.io/issues/54097): an installed-but-unloadable `torchcodec`
kills startup because the guards catch `(ImportError, RuntimeError)` and `torch.ops.load_library`
raises `OSError`. Hits any host without system ffmpeg.

**mergify** (2026-08-27T20:24:08Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @peakcrosser7.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**jschmied** (2026-08-27T21:54:14Z):

**The prebuilt image has drifted behind this branch in a way that fails silently — worth a line in
the PR body.**

`vllm/vllm-openai:qwen38-flash-next` ships `0.1.dev20073+g8e685d198`, and its
`ModelOptMixedPrecisionConfig.get_quant_method()` dispatches:



```
FP8, MXFP8, NVFP4, W4A16_NVFP4
```

This branch (`2a4cd640`) and `main` both dispatch:



```
FP8, FP8_PB_WO, MXFP8, NVFP4, W4A16_NVFP4
```

So the branch is fine. But the image is what the PR body currently points people at, and on the
image a checkpoint declaring `quant_algo: "FP8_PB_WO"` falls through to:



```python
# Layer not in quantized_layers — leave unquantized
return UnquantizedLinearMethod()
```

which loads the **packed FP8 bytes into BF16 parameters**. No error, no warning, server starts
clean, output is fluent garbage. `ModelOptFp8PbWoLinearMethod` is already present in the image’s
`modelopt.py` — only the dispatch branch is missing, so it is a one-line fix locally:



```python
if quant_algo == 'FP8_PB_WO':
  return ModelOptFp8PbWoLinearMethod(self.fp8_config)
```

This is not hypothetical for this model. `lovedheart/Qwen3.8-Flash-Next-NVFP4-FP8` uses
`FP8_PB_WO` (blockwise 128×128, weight-only) for its 156 attention and GDN projection tensors —
i.e. every layer that matters — and it is one of the more attractive Flash-Next checkpoints right
now because it is the only servable one that quantizes the dense weights. Stock SGLang has the
same gap and its author documents it.

A ten-second offline check that would save people the hunt:



```bash
python - <<'PY'
import re, vllm.model_executor.layers.quantization.modelopt as m
s = open(m.__file__).read()
i = s.index("class ModelOptMixedPrecisionConfig"); j = s.index("def get_quant_method", i)
print(sorted(set(re.findall(r'quant_algo == "([A-Z0-9_]+)"', s[j:j+2600]))))
PY
jq -r '.quantization.quantized_layers[].quant_algo' hf_quant_config.json | sort -u
```

The second list must be a subset of the first. Suggestion: either refresh the tag, or add a line
to the PR body noting the image predates `FP8_PB_WO` — a format the runtime does not _recognise_
is much worse than one it rejects, because rejection is an error message and non-recognition is a
silent reinterpretation of the bytes.

---

**Two things from profiling this model on GB10 that may be useful to the thread.**

**1. Single-stream decode is bandwidth-bound on the _unquantized dense_ weights, not on the
experts.** Torch profiler over a steady-state decode, TP=1, no speculation, 58.8 ms/token:

|                               | ms/token | % of wall |
| ----------------------------- | -------: | --------: |
| cuBLAS GEMV (BF16 mat-vec)    |    40.77 |     69.4% |
| cutlass GEMM (the NVFP4 path) |    15.80 |     26.9% |
| MoE experts                   |     0.47 |      0.8% |

GPU is 95.5% busy (union of 253,529 kernel intervals across 52 streams), so this is not a
scheduling or graph-break effect. `RadixArk/Qwen3.8-Flash-Next-NVFP4` leaves \~4.84 B **dense**
params in BF16 — attention q/k/v/o, GDN projections, dense mlp, lm_head, shared_expert — read in
full every token: 9.68 GB of a 10.98 GB per-token budget. At 273 GB/s that is a 35.5 ms floor, and
an independent bandwidth model lands within 15% of the profiler. The properly-NVFP4 experts cost
0.47 ms/token by comparison. An “A6B” model moving 11 GB/token is the headline.

[hashd1ve](https://github.com/hashd1ve/qwen38-flash-next-one-dgx-spark) reached the same
conclusion independently and earlier, from SGLang.

**2. `VLLM_GDN_DECODE_KERNEL=triton` appears to be required once the GDN projections are FP8.**
Reported by `primitive-ai`, who bisected it module class by module class: with the default cuda
kernel, FP8 GDN projections _hang the engine deterministically_ at concurrency \~32 — no error,
requests simply stall. Relevant to anyone serving a dense-quantized checkpoint of this model.

**3. Do not use the published `gsm8k_metrics.json` / `aime26_metrics.json` on these checkpoint
repos to compare variants.** They are byte-identical (sha256 `88766f7e…` / `eb4acd8c…`, same
`latency_seconds` to ten decimals) across a plain NVFP4 build, an FP8 dense-quantized fork of it,
and a 512→448 expert-**pruned** variant, and the `model` field inside them names the original
build. I repeated those figures myself before checking. `sha256sum` them before letting them
inform a decision.

**ssubbotin** (2026-08-28T00:11:29Z):

Tested this branch on a **RTX PRO 6000 Blackwell (SM120, 96 GB), native Linux, TP=1** with `RadixArk/Qwen3.8-Flash-Next-NVFP4`, MTP k=3, `--max-model-len 81920`, `VLLM_PLE_CPU_OFFLOAD=1`. Sharing three things: an FP8 KV result, and two problems that are present on the unmodified branch.

## FP8 KV cache for QSA works, and it is a small change

`Qwen4ExpQSAFlashAttentionBackend` currently declares `["auto", "bfloat16"]`, so Flash-Next is limited to a BF16 pool. Almost everything needed is already in place: `do_kv_cache_update` is inherited and already quantizes through `reshape_and_cache_flash` with the layer scales, `set_default_quant_scales` is already called in the layer constructor, and the pool halves by itself once the spec dtype is FP8, since `AttentionSpec` sizes pages from `get_dtype_size`.

What was missing is the read side: `_qsa_sparse_paged_gqa_splitk_kernel` needs to dequantize on load. The per-tensor key scale folds into the existing `softmax_scale_log2` multiply at no cost, and the value scale applies once to the normalized output, which is exact because a constant commutes through both the softmax normalization and the split-k merge. One detail worth flagging even if you implement this differently: the value dot currently does `probabilities.to(values.dtype)`, which under FP8 would quantize the _probabilities_ rather than the values.

One other thing surprised me. `Qwen4ExpQSAFlashAttentionBackend` inherits `supports_kv_cache_dtype` from `FlashAttentionBackend`, which routes quantized dtypes to `flash_attn_supports_kv_cache_dtype()` and so requires FA3 on SM90 or FA4 on SM100, and refuses `fp8_e5m2` outright; `FlashAttentionImpl.__init__` raises on the same condition. QSA never calls FlashAttention’s attention kernels, so on SM120 FP8 gets refused for a reason unrelated to SM120. Only the arch-independent `reshape_and_cache_flash` write path actually reads the cache dtype.

Measured against a BF16 control booted from the same image in the same session, which reproduced the shipped pool figure exactly:

At `--gpu-memory-utilization 0.97`:

| metric                                 | bf16                  | fp8                       | delta      |
| -------------------------------------- | --------------------- | ------------------------- | ---------- |
| KV memory                              | 10.25 GiB             | 10.25 GiB                 | same       |
| KV pool                                | 280,547 (3.42x @ 80k) | **423,543 (5.17x @ 80k)** | **+51.0%** |
| needle, 4 contexts x 4 depths to \~60k |                       | 16/16                     | exact      |

At `--gpu-memory-utilization 0.93` (see the OOM below for why), the full paired battery:

| metric                            | bf16             | fp8                 | delta      |
| --------------------------------- | ---------------- | ------------------- | ---------- |
| KV pool                           | 176,184 (2.15x)  | **266,675 (3.26x)** | **+51.4%** |
| 8-way mixed prefill+decode        | 180.3 tok/s, 8/8 | 162.6 tok/s, 8/8    | −9.8%      |
| single-stream decode, median of 6 | 149.1 tok/s      | 144.2 tok/s         | −3.3%      |
| MTP acceptance per draft position | 0.530            | 0.519               | −2.1%      |

It is +51% rather than +100% because only the 12 full-attention layers hold FP8 KV; the indexer side caches in `common/qsa_cache.py` stay BF16 and GDN state shares the pool. Deriving from the paired pool sizes, the un-quantized remainder is roughly 32% of the per-token cost, so teaching the side caches the same trick looks worth about another 32% of pool.

Scope of what I tested: CLI-driven `--kv-cache-dtype fp8` with default unit scales. Checkpoint-declared `kv_cache_scheme` stays refused, as do NVFP4 and the int4/int8 per-token-head formats. I patched the AMD mirror for parity but have no AMD hardware to test it on.

Branch, on top of `02f2b4c15d`: <https://github.com/ssubbotin/vllm/tree/qsa-fp8-kv> — 5 files, +350/−28, extending `tests/models/qwen4_exp/test_qsa_reference.py` from 43 to 46 passing. Happy to open it as a PR against `release/qwen38next` if that is useful to you, or to leave it here if you would rather land the model support first.

## Two problems on the unmodified branch

**1. `pytest tests/models/qwen4_exp/` fails collection.**



```
ERROR tests/models/qwen4_exp/test_qsa_amd.py - RuntimeError: Tried to registe...
!!!!! Interrupted: 1 error during collection !!!!!
```

`nvidia/qsa.py` and `amd/qsa.py` both call `direct_register_custom_op("qwen4_exp_qsa_with_output")`, so importing both test modules into one process registers the op twice. Each file passes on its own (`test_qsa_amd.py` gives 10 skipped on NVIDIA). I reproduced this on an unmodified export of `02f2b4c15d` before concluding it was not something I had introduced. It will bite whenever CI collects that directory as a whole.

**2. An 8-way concurrent load OOMs at `--gpu-memory-utilization 0.97`, on BF16 as well as FP8.**

Eight concurrent \~24k-token summarization requests kill the engine on the shipped settings:



```
RuntimeError: Worker failed with error 'CUDA out of memory. Tried to allocate 320.00 MiB.
GPU 0 has a total capacity of 94.97 GiB of which 129.44 MiB is free.
Including non-PyTorch memory, this process has 94.12 GiB memory in use.
Process 726 has 730.00 MiB memory in use.'
```

Process 726 is the PLE offload worker. Its CUDA context costs \~730 MiB on the GPU, and the memory profile does not count it, so at 0.97 there is not enough real headroom left for peak activation under concurrency. Both arms die identically, so this is independent of KV dtype. The engine then shuts down cleanly with `EngineDeadError` and the container exits 0, which reads like a graceful stop rather than a failure. Dropping to 0.93 makes the same 8-way pass complete 8/8 on both arms.

This is the same unaccounted allocation behind the `--kv-cache-memory` suggestion in the startup log advertising more room than exists; it just shows up more sharply under load. Subtracting the offload worker’s context from the profile would fix both symptoms.

**gau-nernst** (2026-08-28T02:10:51Z):

@ssubbotin Thanks for the report on FP8 KV cache. We plan to do some refactoring on the QSA backend after this PR lands in main. We can add FP8 KV cache support after that.

**peakcrosser7** (2026-08-28T03:15:39Z):

/ci run

**github-actions** (2026-08-28T03:15:57Z):

✅ Triggered [Buildkite CI #85921](https://buildkite.com/vllm/ci/builds/85921) for commit `4ab6e99d2464`.

**peakcrosser7** (2026-08-28T04:20:35Z):

/ci run

**github-actions** (2026-08-28T04:20:52Z):

✅ Triggered [Buildkite CI #85925](https://buildkite.com/vllm/ci/builds/85925) for commit `4df2ce22d086`.

**DONGRYEOLLEE1** (2026-08-28T04:40:51Z):

**2× RTX PRO 6000 (SM120) data point: NVFP4 with the FP8 PLE table kept on GPU (no CPU offload), a second engine co-hosted on the same two cards, and confirmation that the prefix-caching `align` promotion fires on the NVIDIA image.**

Setup: 2× RTX PRO 6000 Blackwell Workstation (96 GB, SM120), driver 580.173 / CUDA 13.0, RunPod secure-cloud pod (container without `--privileged`, see [my comment on #53899](https://github.com/vllm-project/vllm/pull/53899#issuecomment-5448518741) for why the offload cannot be used there), image `vllm/vllm-openai:qwen38-flash-next` (`0.1.dev20073+g8e685d198`), `RadixArk/Qwen3.8-Flash-Next-NVFP4`, `--quantization modelopt --tensor-parallel-size 2 --enable-expert-parallel --disable-custom-all-reduce --max-model-len 131072 --max-num-seqs 16 --max-num-batched-tokens 8192 --enable-prefix-caching --no-enable-flashinfer-autotune --reasoning-parser qwen3 --tool-call-parser qwen3_xml --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --compilation-config '{"mode":"NONE","cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[4,8,12,16]}'`, env `VLLM_USE_DEEP_GEMM=0 VLLM_MOE_USE_DEEP_GEMM=0`, BF16 KV. Thanks to net-snix, SirTificate and OsakaTX: the custom-allreduce and BF16-KV findings and the `_get_ple_embedding_quant_method` escape hatch (`VLLM_PLE_FORCE_FP8=1`) are theirs; without the latter the ModelOpt config never reaches the FP8 PLE path and loading stops at `ngram_embedding.weight_scale`.

## 1. NVFP4 + FP8 PLE table resident on GPU

The thread has NVFP4 with offload (SirTificate) and FP8 with the table on GPU (net-snix); this is the remaining combination. Per rank:

|                                                                   | measured                                                            |
| ----------------------------------------------------------------- | ------------------------------------------------------------------- |
| resident weights: NVFP4 experts + BF16 dense + FP8 PLE shard      | 66.1 GiB (40.5 + 25.6)                                              |
| `--gpu-memory-utilization 0.76` → KV pool                         | 239,238 tokens, 1.83× concurrency at 131,072                        |
| weight load / CUDA-graph capture (`FULL_DECODE_ONLY`, sizes 4-16) | 285 s / 7 s, 0.31 GiB                                               |
| single stream, 512-token completions, through the HTTPS proxy     | 122-126 tok/s                                                       |
| MTP k=3 acceptance over \~2.5k drafts (mixed chat + code)         | 51.0 % (per position 69 / 48 / 36 %), 2.53 accepted tokens per step |

Outputs were checked for content (Korean QA with reasoning split by `qwen3`, code generation, a `qwen3_xml` tool call returning `get_weather({"city": "서울"})`), not only for the absence of errors.

## 2. Two engines on the same two cards

The remaining budget runs a second vLLM engine, `Qwen3.8-27B` NVFP4 (compressed-tensors) TP=2 with its own MTP k=3, at `--gpu-memory-utilization 0.19`: 143,945 KV tokens at 65,536, 85-86 tok/s single stream, 54.6 % MTP acceptance. Total 0.95 utilization, 91.1 GB used per card. One thing that cost us a restart: at 0.15 the 27B engine loaded weights and profiled activations to 14.15 GiB and ended with 0.15 GiB for KV, which fails the “at least one max-length request” check (1.15 GiB needed at 65,536); 0.18-0.19 with `--max-num-batched-tokens 4096` is the floor on this card for that pairing.

## 3. Prefix caching: the `align` promotion does fire on this image

Re the dolf3131 (GB10, arm64 image) / zzw09773 (ROCm) disagreement: on `8e685d198` x86_64 with only `--enable-prefix-caching` passed, startup logs



```
Mamba cache mode is set to 'align' for Qwen4ExpForConditionalGeneration by default when prefix caching is enabled
```

so the hybrid-model hook in `model_executor/models/config.py` runs here, and a 1,600-token attention block is what you get. The “inert without `--mamba-cache-mode align`” observation therefore looks specific to the arm64/GB10 image rather than to the branch.

Not verified: the per-token cost of keeping the table on GPU versus offloading it; our tok/s include proxy round-trip and prefill and are not directly comparable to the local 149-161 tok/s numbers above.

**peakcrosser7** (2026-08-28T06:27:20Z):

/ci run nightly

**github-actions** (2026-08-28T06:27:37Z):

✅ Triggered [Buildkite CI #85937](https://buildkite.com/vllm/ci/builds/85937) for commit `0730a29e52cb`.

**mergify** (2026-08-28T06:31:10Z):

Hi @peakcrosser7, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**peakcrosser7** (2026-08-28T06:57:41Z):

/ci run nightly

**github-actions** (2026-08-28T06:58:00Z):

✅ Triggered [Buildkite CI #85941](https://buildkite.com/vllm/ci/builds/85941) for commit `c63d5201649d`.

**ehfd** (2026-08-28T07:59:22Z):

Did anyone test on A100 GPUs?

**peakcrosser7** (2026-08-28T08:57:28Z):

/ci run nightly

**github-actions** (2026-08-28T08:57:45Z):

✅ Triggered [Buildkite CI #85954](https://buildkite.com/vllm/ci/builds/85954) for commit `89d0bb71aeb2`.

**jschmied** (2026-08-28T09:37:51Z):

**`lm_head` cannot be quantized on this model, and one of the three reasons is in this PR’s own
model file — worth a one-line change before merge.**

`lm_head` is BF16 `[248320, 2560]` in every published GPU checkpoint of Flash-Next (verified across
ten). That is not a checkpoint choice. Three independent things block it, and any one alone is
sufficient:

**1. This PR’s model file never passes `quant_config` to `ParallelLMHead`.**

`vllm/models/qwen3_8_flash_next/nvidia/model.py`:



```python
self.lm_head = ParallelLMHead(
  config.vocab_size,
  config.hidden_size,
  prefix=maybe_prefix(prefix, 'lm_head'),  # <- no quant_config
)
```

`VocabParallelEmbedding.__init__` then sees `quant_config is None` and installs
`UnquantizedEmbeddingMethod` unconditionally — `get_quant_method` is never called, so a checkpoint
declaring `lm_head` in `quantized_layers` is silently ignored. `self.quant_config` is already in
scope two lines up, and `ParallelLMHead` already accepts the kwarg:



```python
quant_config = (self.quant_config,)
```

Same in `mtp.py`, which constructs its own `ParallelLMHead` the same way.

**2 and 3 are not this PR’s** — recording them so the picture is complete. `config.json`’s
`quantization_config.ignore` contains `lm_head` in the checkpoints we tested (so `is_layer_excluded`
returns early), and `VocabParallelEmbedding`’s vocab-aware loader asserts
`loaded_weight.shape[output_dim] == self.org_vocab_size`, which is true of `weight` and false of a
block-scale companion shaped `[1940, 20]` — the embedding loader has no notion of a scale tensor.
We worked around the third locally by attaching a plain copy loader to the scale parameters, with
an explicit `NotImplementedError` for TP>1 since the scale would need sharding in _block_ space.

**Why it is worth supporting: measured on one GB10, TP=1.**

|           | `lm_head` BF16 | `lm_head` FP8 blockwise |
| --------- | -------------: | ----------------------: |
| code      |     23.2 tok/s |                **26.1** |
| factual   |     23.7 tok/s |                **26.2** |
| NLL/token |         0.9687 |              **0.9628** |
| tasks     |          10/10 |                   10/10 |

Paired NLL over 14 chunks / 646 tokens of held-out prose, code, German, French and technical text;
9 chunks better and 5 worse, so we read the −0.6% as noise and report it as **no measurable
quality cost**, not as an improvement. **+11% decode** matches a bandwidth model: at `[248320,
2560]` the head is 1.27 GB read on every token, of a \~7.25 GB per-token budget.

Format matters more than the layer: on the sibling Qwen3.8-27B we previously measured an **NVFP4**
head at 2.4% worse NLL and declined it, while an **FP8** head is loss-neutral.

One prefix note for anyone implementing: vLLM asks about **`language_model.lm_head`**, not
`lm_head`, so a config keyed on the bare name still resolves (the candidate list handles it) but is
worth testing explicitly:



```python
c = ModelOptMixedPrecisionConfig.from_config(
  json.load(open('config.json'))['quantization_config']
)
c.is_layer_excluded('language_model.lm_head')  # must be False
c._resolve_quant_algo('language_model.lm_head')  # must be the algo you wrote
```

That ten-second offline check would have saved us three ten-minute boot cycles.

Happy to send the one-line `quant_config` change as a PR against this branch if useful — the other
two are separate and belong upstream of the model.

**abtraore** (2026-08-28T21:51:34Z):

We implemented the FP8 main-KV path this PR leaves gated (`kv_quant_mode` is plumbed but every gate raises “Qwen4Exp QSA requires a BF16 main KV cache”) and validated it on consumer Blackwell. Patch against this PR’s lineage (the `qwen38-flash-next` nightly): <https://gist.github.com/abtraore/329547468a6eb04ecedac38250148093> — happy to turn it into a PR against this branch, or as a follow-up once this merges.

**Design** (mirrors the shipped DeepSeek-V4 `fp8_ds_mla` split of quantized attended-values / dequant-on-read):

- `--kv-cache-dtype fp8` accepted through the five config gates; the parent `FlashAttentionImpl` FA capability probe is bypassed by handing it `"auto"` and restoring the dtype after `super().__init__()` — QSA never runs flash-attn over the cache, so the probe does not apply.
- Write path is unchanged upstream code: the inherited `reshape_and_cache_flash` already quantizes given the dtype string and the layer’s registered scales.
- Read path: the uint8 pages are viewed as `float8_e4m3fn` after the K/V split, and the splitk Triton kernel gains an `IS_FP8: tl.constexpr` — e4m3→bf16 upcast before each `tl.dot` (sm12x cannot dot fp8 operands), K-scale folded into the existing logit scaling, V-scale into the final normalize. Scales are read from the on-device `layer._k_scale/_v_scale` buffers, so CUDA-graph replay stays valid.
- The indexer side-cache stays bf16 (it already ignores `cache_dtype_str`), so block selection precision is untouched — strictly more conservative than DeepSeek’s shipped fp4 indexer cache.

**Validation** (4× RTX 5090 32 GB, sm120, TP=4 + EP, RadixArk NVFP4 checkpoint, MTP n=3; experiment → result):

1. Boot, bf16 baseline → KV pool 389,084 tokens (1.48× at 262,144).
2. Same boot with `--kv-cache-dtype fp8` → pool **692,910 tokens (+78%)**; attention block re-equalizes 800→1600 against the mamba page as expected.
3. Greedy A/B, four prompts, bf16 vs fp8 → final content **md5-identical** on every prompt.
4. Decode: 170 tok/s shallow / 211 on code (≥ bf16’s 169/203); 186–207 tok/s at 115K–224K context. MTP acceptance unchanged (0.68–0.93 across workloads).
5. Prefill at 115–186K: 2,350–2,490 tok/s vs 2,551–2,690 bf16 (−8%, quantize-on-write).
6. Needle retrieval through fp8-scored QSA selection: 3/3 at \~140K (depths 10/50/95%).
7. The larger pool enables what bf16 cannot boot: `--max-model-len 393216` with YaRN 1.5 (bf16 pool 389,084 < 393,216 refuses) → pool 720,159, needle 3/3 at 336K, drafter acceptance 0.74–0.90 past native with `max_model_len` pinned inside `--speculative-config`.

One note for the eventual first-party version: with CPU offload enabled, the PLE offload worker’s CUDA context also breaks the `--kv-cache-memory` boot advisory — filed separately with a fix in [#54275](https://github.com/aarnphm/aarnphm.github.io/issues/54275).

**AI assistance disclosure** (per the contributing policy): developed with AI assistance (Claude Fable 5 via Claude Code). Procedure: kernel-path recon of this PR’s `qsa.py`/`ops/qsa.py`, study of the in-tree DSV4 `fp8_ds_mla` precedent, patch, then the gated validation above on live hardware. The human submitter reviewed the patch and owns the results.

**mergify** (2026-08-28T22:49:30Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @peakcrosser7.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**jschmied** (2026-08-29T17:56:37Z):

**A second `quant_config` hole, and a branch with both fixes applied to this PR’s head.**

`nvidia/hyperconnection.py` hardcodes `quant_config=None` on all three `GatedResidual`
projections (lines 102, 113, 122 at `91a6b555`), so the hyper-connection linears are always
unquantized regardless of what the checkpoint declares. The class docstring twelve lines above
states the opposite:

> …are vLLM Linear modules (merged replicated linear for down+inject), so GEMM dispatch
> (e.g. the low-latency skinny GEMM) applies through the standard `quant_method` mechanism.

It cannot, because `quant_method` is only ever `UnquantizedLinearMethod` for these layers. The
three call sites in `model.py` already have a `quant_config` in scope (the mixer site needs
`vllm_config.quant_config`).

Branch against this PR’s head, rebased onto `91a6b555`, three commits, 12 insertions:
<https://github.com/jschmied/vllm/tree/gb10-sm121-fixes>

1. `GatedResidual` takes and forwards `quant_config`
2. `quant_config` on both `ParallelLMHead` sites — the `lm_head` case from my earlier comment
   (`mtp.py` uses `self.quant_config`, since the drafter resolves its own via
   `get_draft_quant_config()` and the two differ whenever `ignore` carries `mtp.*`)
3. The `MIXED_PRECISION` dispatch gap for `FP8_PER_CHANNEL_PER_TOKEN` — that one also affects
   `main`, so it is filed separately as [#54367](https://github.com/aarnphm/aarnphm.github.io/issues/54367) rather than carried here

**One honest caveat on (1).** We converted 96 hyper-connection `_up` projections to per-channel
FP8 on a GB10 and it was **performance-neutral**: 36.05 vs 36.45 tok/s, n=6 per arm from empty
compile caches, against a measured 6.9% noise floor. A per-shape microbenchmark explains why —
`(10240, 320)` at M≤8 runs 30.8 us against a 24.0 us roofline, i.e. already \~78% of peak, so it
is latency-bound and halving the bytes buys 1.18x; `(336, 10240)` is _slower_ in FP8 (0.89x). By
contrast `(10240, 2560)` gives 2.42x, which is why FP8 pays on the dense projections and not
here.

So (1) is a correctness/reachability fix, not a speed one: today a checkpoint that declares these
layers quantized is silently ignored. I would not want it merged on a performance argument I
cannot support.

**Isotr0py** (2026-08-30T04:44:27Z):

/ci run

**github-actions** (2026-08-30T04:44:45Z):

✅ Triggered [Buildkite CI #86194](https://buildkite.com/vllm/ci/builds/86194) for commit `91a6b555d0b6`.

**ivanium** (2026-08-30T11:26:53Z):

/ci run

**github-actions** (2026-08-30T11:27:11Z):

✅ Triggered [Buildkite CI #86222](https://buildkite.com/vllm/ci/builds/86222) for commit `13c80fb30ab8`.

**ZJY0516** (2026-08-30T14:03:28Z):

> i do still think this should be an MRV2 only model due to the complexity of the MRV1 additions (ideally we want MRV1 to just be stable and in maintenance mode, i.e. not adding new features)

reasonable

**ZJY0516** (2026-08-30T16:56:09Z):

/ci run

**github-actions** (2026-08-30T16:56:27Z):

✅ Triggered [Buildkite CI #86246](https://buildkite.com/vllm/ci/builds/86246) for commit `15c17faca351`.

**ZJY0516** (2026-08-30T18:07:19Z):

/ci run

**github-actions** (2026-08-30T18:07:36Z):

✅ Triggered [Buildkite CI #86250](https://buildkite.com/vllm/ci/builds/86250) for commit `d750ca17f5a8`.

**ivanium** (2026-08-31T00:14:00Z):

/ci run

**github-actions** (2026-08-31T00:14:18Z):

✅ Triggered [Buildkite CI #86262](https://buildkite.com/vllm/ci/builds/86262) for commit `c279484e9695`.

**ZJY0516** (2026-08-31T04:07:44Z):

/ci run

**github-actions** (2026-08-31T04:08:03Z):

✅ Triggered [Buildkite CI #86285](https://buildkite.com/vllm/ci/builds/86285) for commit `b55fb551fd61`.

**tsw2k** (2026-08-31T10:56:14Z):

Running this stack in a configuration you may not have tried: TP4 across four DGX Sparks (GB10, sm121), on the day-0 image. Repo with the launcher, patches and measurements: <https://github.com/tsw2k/Qwen3.8-Flash-Next-Quad-DGX-Sparks>

The good news first: multi-node works. `--nnodes 4` with the mp backend rendezvouses fine, and at native 262k the fleet passes a full gate suite (concurrent \~30k prefills, byte-identical greedy decode, NIAH to 128k, vision, tool calls). 4.7M-token KV pool, 31 tok/s single stream, 157 tok/s aggregate at 16 streams.

Four findings that may be useful for this PR:

1. Plain TP4 cannot load the NVFP4 checkpoint. TP sharding cuts the MoE intermediate 640 to 160 per rank and FLASHINFER_CUTLASS raises `NotImplementedError: Intermediate size padding for w1 and w3`. `--enable-expert-parallel` avoids it. A clearer error message, or implying EP for this case, would save people a boot.

2. `_get_ple_embedding_quant_method` only accepts `Fp8Config`, so `VLLM_PLE_CPU_OFFLOAD=1` with the ModelOpt NVFP4 checkpoint dies loading `ngram_embedding.weight_scale`. The PLE shards are fp8 plus weight_scale either way; accepting `ModelOptNvFp4Config` there fixes it.

3. The PLE offload worker loads the whole table in one process. On a 128 GB GB10 node its load peak reached \~85 GB anon RSS (the fp8 table plus a temporary copy) and drew a global OOM kill at any gpu_memory_utilization we tried. The `nnodes != 1` check in `_validate_ple_offload_config` looks stricter than the architecture requires, since the worker is spawned per WorkerProc and its pidfd handover is node-local, but memory rather than topology is what stops it on this hardware.

4. The open bug: on the 1M YaRN lane, roughly the third request with a prompt over \~100k tokens hangs a worker’s GPU stream. The `sample_tokens` RPC times out and the engine dies. `/health` is 200 until then, and short requests never trigger it. We eliminated, one variable per boot on a rebooted fleet: prefix caching, MTP, expandable_segments, page cache (an unconditional flusher through serving), chunk size 8192 down to 1024, CUDA graphs (`--enforce-eager` dies identically), and three different PLE paths (the blazux mmap patch, the resident vocab-sharded table, and CPU offload as far as it would go). py-spy shows every surviving rank parked at the first stream-sync point after the wedge, so the hang is in a kernel enqueued earlier in the forward: GDN, QSA or the EP all-to-all. Full isolation matrix and stack: <https://github.com/tsw2k/Qwen3.8-Flash-Next-Quad-DGX-Sparks/blob/main/docs/OPEN-PROBLEMS.md>

Happy to run instrumented boots or share full logs if that helps.

**tsw2k** (2026-08-31T20:02:25Z):

One more data point for finding 4: the same failure shape has now been reported on a different engine. A withdrawn 2x GB10 SGLang profile for this checkpoint hit invalid sampling probabilities with CUDA asserts and Xid 43 under multi-turn agentic traffic, with the same trigger sequence of long prefill, decode, tool turn, longer prefill: <https://forums.developer.nvidia.com/t/381836>

Two engines, one hardware platform, one traffic shape. Our runs add that the hang survives MTP=0, so speculative decoding is not required to trigger it. Together this points below the engine layer, toward shared kernels or numerics on sm121.

**hnhyzz** (2026-09-17T09:19:21Z):

Flash Attention is not required here. Why do we reply on FlashAttn only for its KV cache update?

**gau-nernst** (2026-09-17T09:21:18Z):

@hnhyzz Yea we don’t need Flash Attention subclass, I plan to remove it in a future PR. Or you can submit a PR for that as well, much appreciate.

**hnhyzz** (2026-09-17T14:55:27Z):

> @hnhyzz Yea we don’t need Flash Attention subclass, I plan to remove it in a future PR. Or you can submit a PR for that as well, much appreciate.

I deleted the dependency to flash attention. The Qwen4ExpQSABackend and Qwen4ExpQSAImpl now inherent AttentionBackend and AttentionImpl.
@gau-nernst Could you please check?
[vllm-project/vllm#57395](https://github.com/vllm-project/vllm/pull/57395)

