---
title: '51274'
pageLayout: default
slug: thoughts/inference/vllm-prs/51274
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/51274.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#51274](https://github.com/aarnphm/aarnphm.github.io/issues/51274) \[ROCm\]\[Kimi-K3\] Add opt-in gfx942 MXFP4-to-int4 conversion

[vllm-project/vllm#51274](https://github.com/vllm-project/vllm/pull/51274) · maeehart · MERGED · created 2026-08-06T15:09:53Z · merged 2026-10-03T06:28:02Z

## Summary

Add an explicit `int4_per_group_32` MoE override for Kimi-K3 on gfx942. When selected, the MXFP4 expert weights are requantized once at load time and served through AITER’s replacement BF16 x packed-int4 FlyDSL path.

The conversion is lossy and never happens by architecture alone. The default MXFP4 path is unchanged.

## Why

gfx942 does not support the scaled FP4 conversion instructions used by the native MXFP4 path. This opt-in path pays the conversion cost once during model loading and then uses the replacement packed-int4 kernels from [ROCm/aiter#4646](https://github.com/ROCm/aiter/pull/4646).

## Changes

- Register `int4_per_group_32` as a MoE weight quantization key.
- Require the explicit `--quantization-config.moe.weight int4_per_group_32` override on gfx942.
- Require the replacement A16W4/int4 AITER entry point and fail closed unless it supports SiTUv2.
- Convert eight experts at a time and release each source tensor before converting the next one, bounding transient memory during load.
- Log a warning that the selected conversion is lossy.
- Leave `int4_per_group_32` out of the online MoE table so `Mxfp4MoEMethod` can convert at load time.

## Validation

Unit tests:



```text
tests/quantization/test_quantization_config_args.py
tests/models/kimi_k3/test_gfx942_int4.py
```

Strict AITER comparison against the torch reference passed for the production EP shape:



```text
model_dim=3584
intermediate_dim=3072
local_experts=112
topk=15
tokens=1,4,7,16,32
activation=SiTUv2
beta=4.0
linear_beta=25.0
```

Re-checked on 8x MI325X with TP8, expert parallelism, Triton RMSNorm, Triton MLA, AITER fused MoE, and full plus piecewise graph capture. Image: `vllm/vllm-openai-rocm:nightly-dc36fcce902a63eab06c1b93a5c4a5ee178a0c56` (AITER v0.1.21.post2, picked up in [#55968](https://github.com/aarnphm/aarnphm.github.io/issues/55968)). Serving used the replacement packed-int4 path from [ROCm/aiter#4646](https://github.com/ROCm/aiter/issues/4646). Current FlyDSL 0.3.2 `gemm2` cannot select `BUFFER_ATOMIC_FADD` of `v2bf16` on gfx942, so those runs used the `moe_2stage_a16wmix` kernels from [ROCm/aiter#4646](https://github.com/ROCm/aiter/issues/4646) on top of AITER v0.1.21.post2.

- The int4 path was FlyDSL A16W4 SiTUv2 (`ActivationType.Situv2`, `torch.int4`, `QuantType.per_1x32`, `flydsl_moe1_abf16_wint4_*`). Untuned `a16wi4` is still this path. Graph capture completed. MXFP4 native and emulation were not used.
- Chat completions returned coherent answers, including a Fibonacci Python snippet.
- 1M NIAH passed at 131072, 524288, and [`1047552`](https://github.com/aarnphm/aarnphm.github.io/commit/1047552) tokens.
- DSpark k=7 (`Inferact/Kimi-K3-DSpark`) at 262K answered Paris, completed random chat through 250000 tokens, and passed NIAH at 65536, 131072, and 250000 with 512 output tokens. After GSM8K, 104973 of 241563 draft tokens were accepted (43.45%).
- Five-shot GSM8K through `lm_eval` `local-completions` (`num_fewshot=5`, `temperature=0.0`, `max_gen_toks=512`) scored 0.9598 strict-match and 0.9598 flexible-extract on all 1,319 questions, stderr 0.0054. That is within noise of the 0.9560 A16W4 baseline (stderr 0.0056) on the same 1,319 questions. The DSpark GSM8K run matched that 0.9598 number.
- The matched non-DSpark 262K NIAH passed the same three 512-token needles.
- `vllm bench serve` 1K input / 128 output completed 10/10 requests at concurrency 1 and 160/160 requests at concurrency 16, with no server errors. Those serving runs used the untuned FlyDSL a16w4 fallback, so they are a functional check, not a tuned peak.

## Dependency and scope

[ROCm/aiter#4646](https://github.com/ROCm/aiter/issues/4646) is in AITER v0.1.21.post2. vLLM picked that tag up in [#55968](https://github.com/aarnphm/aarnphm.github.io/issues/55968), so this branch is no longer blocked on the AITER release.

The conversion is an explicit opt-in for deployments that accept a lossy one-time weight conversion. It does not change the default checkpoint path.

A follow-up can add `Mxfp4MoeBackend.AITER_INT4_BF16`. This PR stays on the measured FlyDSL SiTU path. It does not wait on [#55684](https://github.com/aarnphm/aarnphm.github.io/issues/55684).

This PR no longer changes `vllm/config/speculative.py`. [#56930](https://github.com/aarnphm/aarnphm.github.io/issues/56930) already keeps expert parallel off for non-MoE drafts such as DSpark when the target uses `--enable-expert-parallel`.

## Test plan

- [x] Quantization key and opt-in selection unit tests.
- [x] Strict production-shape comparison against the torch reference.
- [x] Full 1,319-question GSM8K after [ROCm/aiter#4646](https://github.com/ROCm/aiter/issues/4646).
- [x] Target-only graph capture and serving sanity.
- [x] Long-context and DSpark validation against the AITER release containing [ROCm/aiter#4646](https://github.com/ROCm/aiter/issues/4646).
- [x] ROCm CI on the rebased branch.

---

## comments

**maeehart** (2026-08-06T15:11:06Z):

I have read the DCO document and hereby sign off past commits made by me.

**maeehart** (2026-08-12T20:42:42Z):

I have read the DCO document and hereby sign off on the commits in this pull request.

**mergify** (2026-08-14T21:24:18Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maeehart.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-02T13:25:51Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maeehart.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**xaguilar-amd** (2026-09-04T07:22:31Z):

@maeehart AITER has been bumped to 0.1.21.post1 [vllm-project/vllm#52826](https://github.com/vllm-project/vllm/pull/52826) we can push forward this one now.

**fxmarty-amd** (2026-09-09T16:59:32Z):

Hi @maeehart, do you think this could use [vllm-project/vllm#55684](https://github.com/vllm-project/vllm/pull/55684) ?

This would involve:

- Writing `Mxfp4MoEMethod.dequantize_weight`
- Extending [vllm-project/vllm#55684](https://github.com/vllm-project/vllm/pull/55684) to MOE methods.
- Adding int4 online MOE method to: <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization/online> (similar to `Int8OnlineMoEMethod`)

**mergify** (2026-09-10T15:31:39Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maeehart.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**maeehart** (2026-09-10T17:40:50Z):

@fxmarty-amd I looked at routing this through [#55684](https://github.com/aarnphm/aarnphm.github.io/issues/55684). I would keep this PR on the current `Mxfp4MoEMethod` load-time conversion.

[#55684](https://github.com/aarnphm/aarnphm.github.io/issues/55684) is still open and linear-only. `resolve_quant_method` still rejects online requant of a pre-quantized MoE layer. A full `Mxfp4MoEMethod.dequantize_weight` to BF16 also does not fit. The MXFP4 unpack is about 8x the packed weight and peaks above 20 GiB per rank, which is why this PR converts eight experts at a time.

The decode path we measured is AITER FlyDSL a16w4 with SiTUv2 through `AiterExperts`. `Int8OnlineMoEMethod` selects Triton/Marlin/CPU. `fused_flydsl_moe` hardcodes SiLU. Either would drop the activation this model needs.

Five-shot GSM8K on this path is 0.9598 strict-match on 1,319 questions, stderr 0.0054.

MoE requant through the online API can follow once [#55684](https://github.com/aarnphm/aarnphm.github.io/issues/55684) covers MoE. I do not want to block this opt-in gfx942 path on that.

**maeehart** (2026-09-10T17:40:52Z):

@dllehr-amd Can you review this for merge if you agree with keeping the current `Mxfp4MoEMethod` load-time path?

Five-shot GSM8K held at 0.9598 strict-match on 1,319 questions, stderr 0.0054. I replied to @fxmarty-amd on the online-requant thread and to @simondanielsson on the inline comments. The leftover duplicate SiTU entry is removed in [`6a0a733`](https://github.com/aarnphm/aarnphm.github.io/commit/6a0a73312).

Mergify currently reports merge conflicts with main, so a rebase is still needed before merge.

**fxmarty-amd** (2026-09-10T18:07:45Z):

> MoE requant through the online API can follow once [vllm-project/vllm#55684](https://github.com/vllm-project/vllm/pull/55684) covers MoE. I do not want to block this opt-in gfx942 path on that.

yes, makes sense, we are still a bit far from it, I agree.

> The decode path we measured is AITER FlyDSL a16w4 with SiTUv2 through AiterExperts. Int8OnlineMoEMethod selects Triton/Marlin/CPU. fused_flydsl_moe hardcodes SiLU. Either would drop the activation this model needs.

Well, this is the reason why oracles / MOE backends and abstractions as <https://github.com/vllm-project/vllm/blob/8359e15aae32dee9dc1f259a9b2574fb72b5507e/vllm/model_executor/layers/fused_moe/modular_kernel.py#L628-L634> were introduced.

IMO ideally, this should make use of <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/oracle/int_wna16.py> that is supposed to handle MOE backends for `kInt4Static32` weights. I am not sure it makes sense to have this as part of MXFP4 oracle, that is normally meant to handle backends making use of MXFP4 weights?

The MOE oracle refactor was meant to separate the:

- weight creation,
- backend selection,
- weight processing to be mapped to what backends except

and having the re-quantization step, custom backend selection & processing all under `mxfp4.py` here sounds like we are not adopting these oracle/backend abstractions.

I don’t think this is blocking, just nit comment!

**maeehart** (2026-09-10T18:37:48Z):

I have read the DCO document and hereby sign off on the commits in this pull request.

**maeehart** (2026-09-10T18:39:01Z):

I have read the DCO document and hereby sign off on the commits in this pull request.

**maeehart** (2026-09-10T18:39:02Z):

@fxmarty-amd Agreed. I would keep this PR as the load-time special case.

`int_wna16.py` currently selects Marlin, Triton, FlashInfer TRTLLM, Humming, CPU, XPU, and emulation. It has no AITER backend. The measured path is `AiterExperts` FlyDSL a16w4 with SiTUv2. Routing the converted weights through that oracle would drop the activation.

The checkpoint is still MXFP4. The int4 tensors exist only after `_setup_kernel_k3_situ_gfx942`. That is why the convert and kernel pick still live in `Mxfp4MoEMethod` for this PR.

A follow-up can add AITER packed-int4 to the WNA16 oracle, or add `Mxfp4MoeBackend.AITER_INT4_BF16` as in the inline comment. I do not want to do that rewrite in this PR.

**maeehart** (2026-09-11T14:50:51Z):

Can you get this reviewed @dllehr-amd. We have discussed this PR also separately and the design concerns are now resolved.

**maeehart** (2026-09-14T20:36:20Z):

I reran gfx942 MI325X validation on this branch, then landed two follow-up commits and merged current main. Head is [`58c2a37`](https://github.com/aarnphm/aarnphm.github.io/commit/58c2a374).

The int4 path was FlyDSL A16W4 SiTUv2 (`ActivationType.Situv2`, `torch.int4`, `QuantType.per_1x32`, `flydsl_moe1_abf16_wint4_*`, untuned a16wi4 fallback). Graph capture completed. MXFP4 native and emulation were not used.

16K Fibonacci was coherent. 1M NIAH passed at 128K, 512K, and 1,047,552 tokens. DSpark k=7 at 262K answered Paris, completed random chat through 250K, and passed NIAH at 65,536 / 131,072 / 250,000 with 512 output tokens. DSpark GSM8K 5-shot on 1,319 questions was strict-match 0.9598. The matched non-DSpark 262K NIAH also passed.

`3db224c` keeps `--quantization-config.moe.weight int4_per_group_32` out of the online MoE table so `Mxfp4MoEMethod` can convert at load time. `f55575f` keeps expert parallel off for non-MoE drafts so DSpark can start when the target uses `--enable-expert-parallel`.

Current nightly FlyDSL 0.3.2 gemm2 cannot select BUFFER_ATOMIC_FADD of v2bf16 on gfx942. Serving used the moe_2stage_a16wmix kernels from [ROCm/aiter#4646](https://github.com/ROCm/aiter/issues/4646) on top of AITER 0.1.21.post2.

Lightweight checks already pass. Full AMD CI still needs the `ready` label, then `/amd-ci run`. Can you add `ready` and review for merge if this looks right, @dllehr-amd?

**mergify** (2026-09-17T01:24:47Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maeehart.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**maeehart** (2026-09-17T16:31:37Z):

Merged current main. Head is [`12537cc`](https://github.com/aarnphm/aarnphm.github.io/commit/12537cc9).

The only conflict was `vllm/config/speculative.py`. I took [#56930](https://github.com/aarnphm/aarnphm.github.io/issues/56930)’s `draft_model_config` expert-parallel handling and dropped our `draft_is_moe` helper, which that PR already covers. Also renamed `is_fused_moe_situv2_a8w4_enabled` to `is_fused_moe_situv2_enabled` after [#53940](https://github.com/aarnphm/aarnphm.github.io/issues/53940).

**github-actions** (2026-09-23T15:12:01Z):

✅ @maeehart, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**maeehart** (2026-09-23T16:28:07Z):

Trimmed comments. New commit is `1e219231cbbf`. Merged current main.

The expert-parallel bullet was leftover. [#56930](https://github.com/aarnphm/aarnphm.github.io/issues/56930) already keeps EP off for non-MoE drafts such as DSpark, so this PR no longer changes `vllm/config/speculative.py`.

**maeehart** (2026-09-23T16:28:09Z):

/ci run

**maeehart** (2026-09-23T16:28:10Z):

/amd-ci run

**github-actions** (2026-09-23T16:29:15Z):

❌ This PR is 1 commit behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5798604490 -->

**github-actions** (2026-09-23T16:29:34Z):

❌ This PR is 1 commit behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/amd-ci run`. To test this branch at your own risk, use `/amd-ci run --allow-stale`.

<!-- vllm-ci-command:5798604780 -->

**maeehart** (2026-09-23T16:48:32Z):

/ci run

**maeehart** (2026-09-23T16:48:33Z):

/amd-ci run

**github-actions** (2026-09-23T16:48:52Z):

✅ Triggered [Buildkite CI #90713](https://buildkite.com/vllm/ci/builds/90713) for commit `c29f51aa8d8f`.

**github-actions** (2026-09-23T16:49:12Z):

✅ Triggered [Buildkite AMD CI #13482](https://buildkite.com/vllm/amd-ci/builds/13482) for commit `c29f51aa8d8f`.

**maeehart** (2026-09-23T17:41:46Z):

/ci run

**maeehart** (2026-09-23T17:41:48Z):

/amd-ci run

**github-actions** (2026-09-23T17:42:07Z):

✅ Triggered [Buildkite CI #90723](https://buildkite.com/vllm/ci/builds/90723) for commit `ca3ddef9aaed`.

**github-actions** (2026-09-23T17:42:30Z):

✅ Triggered [Buildkite AMD CI #13486](https://buildkite.com/vllm/amd-ci/builds/13486) for commit `ca3ddef9aaed`.

**maeehart** (2026-09-23T20:23:39Z):

The remaining red checks are not test failures from this change.

19 H200 jobs failed at 18:03 UTC in the agent environment hook, before pytest:



```
aws: [ERROR]: Could not connect to the endpoint URL: "https://api.ecr-public.us-east-1.amazonaws.com/"
password is empty
```

`nvidia-h200-mig-18gb-basic-correctness` then timed out at 20:21 UTC on `h200-ci-5` (exit -1). Buildkite cancelled it while `test_cpu_offload[True-False]` was still starting. The previous case, `test_cpu_offload[False-False]`, had already passed. That log has no FAILED test.

AMD CI build 13486 passed. All five `amd-mi300-moe-kernels` shards passed, including shard 2.

The PR is approved. Could you merge it with these CI failures @dllehr-amd, @tjtanaa, or somebody? I have not touched those code paths.

**maeehart** (2026-09-24T15:22:48Z):

/ci run

**github-actions** (2026-09-24T15:23:57Z):

✅ Triggered [Buildkite CI #90962](https://buildkite.com/vllm/ci/builds/90962) for commit `a01889640c6d`.

**maeehart** (2026-09-28T17:19:01Z):

The `/ci run` on `a01889640c6d` ([Buildkite #90962](https://buildkite.com/vllm/ci/builds/90962)) failed again only in NVIDIA H200 MIG jobs. None of the 11 failures comes from this change:

- `entrypoints-integration-ll`, `pytorch-fullgraph`, `quantization-shard-2`, and `v1-kv-connectors-shard-4` stopped in the agent environment hook, before any test, with `aws: [ERROR]: Could not connect to the endpoint URL: "https://api.ecr-public.us-east-1.amazonaws.com/"`.
- `quantization-shard-1`, `v1-kv-connectors-shard-1`, and `v1-kv-offload` failed in `uv pip install` because they could not connect to `download.pytorch.org` or `pypi.org`.
- `engine-core`, `entrypoints-integration-mu`, and `lm-eval-small-models` could not download test images or the gsm8k data from `vllm-public-assets.s3.us-west-2.amazonaws.com`. The errors are DNS failures and read timeouts.
- `regression` logged `Temporary failure in name resolution`. Then `test_gc` failed because its engine found only 1.2 of 16.0 GiB free on the GPU at startup. The engine starts before and after it in the same job found 15.89 GiB free.

All 100 AMD jobs in the same build passed, including `amd-mi355-kimi-k3` and the five `amd-mi300-moe-kernels` shards.

The PR is approved and has the `ready` label. Could you retry the failed jobs or merge it past them, @dllehr-amd or @tjtanaa?

**maeehart** (2026-09-29T14:08:33Z):

/ci retry

**github-actions** (2026-09-29T14:09:02Z):

✅ Triggered [Buildkite CI #91870](https://buildkite.com/vllm/ci/builds/91870) for commit `8ba04ec7e932`, running 12 failed step(s) from [Buildkite CI #90962](https://buildkite.com/vllm/ci/builds/90962).

**maeehart** (2026-09-29T17:32:26Z):

/ci retry

**github-actions** (2026-09-29T17:32:45Z):

✅ Queued 1 failed job(s) for retry in [Buildkite CI #91870](https://buildkite.com/vllm/ci/builds/91870).

**maeehart** (2026-09-29T19:42:40Z):

I re-checked head `8ba04ec7e932` after the `main` merge.

[Buildkite CI #91870](https://buildkite.com/vllm/ci/builds/91870) passed on this commit. It reran the 12 NVIDIA H200 MIG steps that did not pass in [#90962](https://buildkite.com/vllm/ci/builds/90962). One job, `Entrypoints Integration (API Server) 1`, exited after 3 seconds on its first try and passed on retry. All 100 `amd-*` jobs had already passed in [#90962](https://github.com/aarnphm/aarnphm.github.io/issues/90962) on `a01889640c6d`, including `amd-mi355-kimi-k3`, which runs `tests/models/kimi_k3`, and the five `amd-mi300-moe-kernels` shards. The PR diff has not changed since then. The merge only brought in `main`.

Two changes on `main` since [#90962](https://github.com/aarnphm/aarnphm.github.io/issues/90962) touch the gfx942 int4 serve, so I ran it again on 8x MI325X.

- [#58201](https://github.com/aarnphm/aarnphm.github.io/issues/58201) makes `a4w4` the SiTUv2 default, so vLLM now exports `AITER_SITUV2_A4W4=1`. AITER v0.1.23 checks for int4 weights before it reads that variable, so it still uses bf16 activations. The gate/up layout also stays separated, which is what this PR writes. All 416 fused MoE entries in the AITER log have `torch.bfloat16` activations, `torch.int4` weights, and `QuantType.per_1x32`. The kernels that ran are `flydsl_moe1_abf16_wint4_bf16_t16x128x128` and `flydsl_moe2_abf16_wint4_bf16_t16x128x128_atomic`, plus the `t32x128x128` pair.
- [#54956](https://github.com/aarnphm/aarnphm.github.io/issues/54956) turns on the sharded latent MoE up-projection under expert parallel. This serve took that path and logged `Kimi-K3 latent-MoE tail: up-projecting only this rank's hidden shard into the shared output.`

The run used `vllm/vllm-openai-rocm:nightly-36768d1bfd39094681cdbc8cb37d4b31c0729c89` (AITER v0.1.23, FlyDSL 0.3.4.1) with the three changed source files from this PR copied on top. `VLLM_ROCM_USE_AITER_MOE_SITUV2` was unset. The serve used TP8, expert parallel, and `FULL_AND_PIECEWISE` CUDA graphs, and graph capture finished. The AITER kernel overlay from the earlier runs is no longer needed, because AITER v0.1.23 has the gfx942 path in the a16w4 `gemm2`. The 31 tests in the two test files this PR changes pass on gfx942.

Five-shot GSM8K on all 1,319 questions scored 0.9621 ± 0.0053 on both strict-match and flexible-extract. The runs on 2026-09-09 and 2026-09-14 scored 0.9598, so the difference is less than one standard error. The server logged no errors during the run.

This is ready to merge from my side. @dllehr-amd @tjtanaa, can one of you merge it?

**dllehr-amd** (2026-09-30T15:33:47Z):

/ci retry

**github-actions** (2026-09-30T15:34:58Z):

✅ No failed, timed-out, or expired jobs need retrying in [Buildkite CI #91870](https://buildkite.com/vllm/ci/builds/91870).

**dllehr-amd** (2026-09-30T15:37:37Z):

/ci run

**github-actions** (2026-09-30T15:37:58Z):

❌ This PR is 50 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5914559980 -->

**dllehr-amd** (2026-09-30T15:44:40Z):

/ci run

**github-actions** (2026-09-30T15:45:05Z):

✅ Triggered [Buildkite CI #92189](https://buildkite.com/vllm/ci/builds/92189) for commit `9687b05949f0`.

**vllm-agent** (2026-09-30T16:08:47Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 67 test steps (132 jobs) instead of 78 (108 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 78 (108)      | 67 (132) | 41 (46)    | 30 (70)   |
| AMD mirrors            | 81 (123)      | 47 (100) | 50 (58)    | 16 (35)   |

<details><summary>Selector would run (67)</summary>

- `amd-fp8-moe-kernels-mi355`
- `amd-native-quantization-kernels-mi355`
- `arm-cpu-test` ×3
- `async-engine-inputs-utils-worker`
- `basic-models-test-other-cpu`
- `basic-models-tests-extra-initialization` ×14
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `batch-invariance-b200`
- `cpu-distributed-tests-dp-tp`
- `cpu-distributed-tests-pp-tp`
- `cpu-kernel-tests` ×2
- `cpu-language-generation-and-pooling-model-tests` ×3
- `cpu-multi-modal-model-tests-n` ×4
- `cpu-multimodal-config`
- `cpu-params-env-tokenizers-parser`
- `cpu-quantization-model-tests`
- `cpu-qwen2-5-vl-multimodal-tests`
- `cpu-reasoning-renderers`
- `cpu-spec-decode-tests`
- `deepseek-v4-kernel-test-b200`
- `deepseek-v4-kernel-test-h100`
- `distributed-dp-tests-4-gpus`
- `distributed-model-tests-2-gpus` ×3
- `distributed-torchrun-examples-4-gpus`
- `engine`
- `entrypoints-integration-pooling`
- `fusion-e2e-tp2-b200`
- `gemm-rs-ar-2xb200`
- `inkling-unit-tests-b200`
- `kernels-attention-test` ×7
- `kernels-b200` ×3
- `kernels-core-operation-test` ×3
- `kernels-deepgemm-test-h100`
- `kernels-flashmla-test-h100`
- `kernels-fusedmoe-layer-test-2-b200s`
- `kernels-fusedmoe-layer-test-2-h100s`
- `kernels-helion-test` ×5
- `kernels-mamba-test`
- `kernels-mhc-test-b200`
- `kernels-moe-test` ×5
- `kernels-quantization-test` ×6
- `kernels-root-misc-test-b200`
- `kimi-k3-prefix-cache-4xb200` ×2
- `kimi-k3-unit-tests-b200`
- `kv-offload-large`
- `language-models-tests-extra-standard` ×2
- `language-models-tests-standard`
- `lora` ×4
- `model-executor`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `plugin-tests-2-gpus`
- `pytorch-compilation-unit-tests`
- `quantization` ×4
- `quantized-models-test`
- `quantized-moe-test-b200`
- `qwen4-exp-unit-tests-cpu`
- `samplers-test`
- `v1-attention-b200` ×2
- `v1-attention-h100-mi300` ×2
- `v1-kv-connectors` ×4
- `v1-logits-oracle`
- `v1-metrics-lmeval`
- `v1-others-cpu`
- `v1-spec-decode`

</details>

<details><summary>Would skip (today's rules run them) (41)</summary>

- `amd-lm-eval-small-models-harness`
- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `batch-invariance-h100`
- `benchmarks-cli-test`
- `cpu-tool-parsers`
- `distributed-compile-unit-tests-2xh100`
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `fusion-and-compile-unit-tests-2xb200`
- `fusion-e2e-quick-h100`
- `fusion-e2e-tp2-quick-h100`
- `kernels-fla-ops-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `lm-eval-small-models`
- `metrics-tracing-2-gpus`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `pytorch-compilation-dynamic-shapes`
- `pytorch-compilation-passes-unit-tests`
- `pytorch-compilation-unit-tests-h100`
- `pytorch-fullgraph-cudagraph-l4-compatibility`
- `pytorch-fullgraph-test`
- `regression`
- `samplers-multimodal-beam-search`
- `v1-core`
- `v1-executor-worker`
- `v1-kv-offload`
- `v1-sample`

</details>

<details><summary>Would add (today's rules do not run them) (30)</summary>

- `amd-fp8-moe-kernels-mi355` (code map)
- `arm-cpu-test` ×3 (code map)
- `basic-models-tests-extra-initialization` ×14 (code map)
- `cpu-distributed-tests-dp-tp` (code map)
- `cpu-distributed-tests-pp-tp` (code map)
- `cpu-kernel-tests` ×2 (code map)
- `cpu-multi-modal-model-tests-n` ×4 (code map)
- `cpu-quantization-model-tests` (code map)
- `cpu-qwen2-5-vl-multimodal-tests` (code map)
- `cpu-spec-decode-tests` (code map)
- `deepseek-v4-kernel-test-b200` (code map)
- `deepseek-v4-kernel-test-h100` (code map)
- `distributed-dp-tests-4-gpus` (code map)
- `distributed-torchrun-examples-4-gpus` (Python record)
- `gemm-rs-ar-2xb200` (code map)
- `inkling-unit-tests-b200` (code map)
- `kernels-attention-test` ×7 (code map)
- `kernels-b200` ×3 (code map)
- `kernels-core-operation-test` ×3 (code map)
- `kernels-flashmla-test-h100` (code map)
- `kernels-helion-test` ×5 (code map)
- `kernels-mamba-test` (code map)
- `kimi-k3-prefix-cache-4xb200` ×2 (code map)
- `kv-offload-large` (Python record)
- `language-models-tests-extra-standard` ×2 (code map)
- `lora` ×4 (code map)
- `plugin-tests-2-gpus` (code map)
- `qwen4-exp-unit-tests-cpu` (code map)
- `v1-attention-b200` ×2 (code map)
- `v1-attention-h100-mi300` ×2 (code map)

</details>

<details><summary>AMD mirrors: would skip (50)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `batch-invariance-h100`
- `benchmarks-cli-test`
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `fusion-and-compile-unit-tests-2xb200`
- `fusion-e2e-quick-h100`
- `fusion-e2e-tp2-ar-rms-config-sweep-h100`
- `fusion-e2e-tp2-b200`
- `fusion-e2e-tp2-quick-h100`
- `kernels-fla-ops-test-b200`
- `kernels-fusedmoe-layer-test-2-b200s`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `lm-eval-small-models`
- `metrics-tracing-2-gpus`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `openai-api-correctness`
- `pipeline-context-parallelism-4-gpus`
- `platform-tests`
- `pytorch-compilation-dynamic-shapes`
- `pytorch-compilation-passes-unit-tests`
- `pytorch-compilation-unit-tests-h100`
- `pytorch-fullgraph-test`
- `regression`
- `samplers-multimodal-beam-search`
- `spec-decode-draft-model` ×4
- `spec-decode-eagle-1-deepseek-qwen`
- `spec-decode-eagle-2-llama3-qwen-vl-other`
- `spec-decode-mtp-deepseek-mimo`
- `spec-decode-mtp-gemma4`
- `spec-decode-mtp-qwen3-5`
- `spec-decode-ngram-suffix`
- `spec-decode-speculators`
- `v1-core`
- `v1-executor-worker`
- `v1-kv-offload`
- `v1-sample`

</details>

<details><summary>AMD mirrors: would add (16)</summary>

- `deepseek-v4-kernel-test-h100` (code map)
- `distributed-torchrun-examples-4-gpus` (Python record)
- `engine-1-gpu` (code map)
- `inkling-unit-tests-b200` (code map)
- `kernels-attention-test` ×7 (code map)
- `kernels-b200` ×3 (code map)
- `kernels-core-operation-test` ×3 (code map)
- `kernels-flashmla-test-h100` (code map)
- `kernels-helion-test` ×5 (code map)
- `kernels-mamba-test` (code map)
- `kv-offload-large` (Python record)
- `lora` ×4 (code map)
- `plugin-tests-2-gpus` (code map)
- `qwen4-exp-unit-tests` (code map)
- `v1-attention-b200` ×2 (code map)
- `v1-attention-h100-mi300` ×2 (code map)

</details>

<sub>5 changed files · base `111f71a6db` · head `74bc1d2b3f` · Python record: build 92561 at `4056c8ac1f` · kernel record: table [`4056c8a`](https://github.com/aarnphm/aarnphm.github.io/commit/4056c8ac1f) (build 92561), map [`4056c8a`](https://github.com/aarnphm/aarnphm.github.io/commit/4056c8ac1f) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 23 optional steps the selector would also run</sub>

**maeehart** (2026-09-30T19:47:24Z):

/ci retry

**github-actions** (2026-09-30T19:47:44Z):

✅ Queued 1 failed job(s) for retry in [Buildkite CI #92189](https://buildkite.com/vllm/ci/builds/92189).

**maeehart** (2026-09-30T22:05:58Z):

/ci retry

**github-actions** (2026-09-30T22:06:20Z):

✅ Triggered [Buildkite CI #92259](https://buildkite.com/vllm/ci/builds/92259) for commit `2023373524ca`, running 1 failed step(s) from [Buildkite CI #92189](https://buildkite.com/vllm/ci/builds/92189).

**maeehart** (2026-10-01T18:17:26Z):

/ci run

**github-actions** (2026-10-01T18:17:49Z):

✅ Triggered [Buildkite CI #92466](https://buildkite.com/vllm/ci/builds/92466) for commit `503af471e876`.

**maeehart** (2026-10-01T18:18:25Z):

@dllehr-amd I started [Buildkite CI #92466](https://buildkite.com/vllm/ci/builds/92466) on `503af471e876`. Could you turn on auto-merge (squash) for this PR, so it merges when that build passes?

Required CI passed twice already, on `8ba04ec7e932` ([#91870](https://buildkite.com/vllm/ci/builds/91870)) and on `2023373524ca` ([#92259](https://buildkite.com/vllm/ci/builds/92259)). Each time, Mergify merged `main` into the branch before anyone merged the PR, and the new commit had no `buildkite/ci/pr` result. The lines this PR adds and removes have not changed since Sep 24.

**maeehart** (2026-10-02T14:20:01Z):

/ci retry

**github-actions** (2026-10-02T14:21:01Z):

✅ Queued 1 failed job(s) for retry in [Buildkite CI #92466](https://buildkite.com/vllm/ci/builds/92466).

**mergify** (2026-10-02T20:12:42Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maeehart.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**maeehart** (2026-10-02T20:27:56Z):

/ci run

**github-actions** (2026-10-02T20:29:02Z):

✅ Triggered [Buildkite CI #92699](https://buildkite.com/vllm/ci/builds/92699) for commit `74bc1d2b3f61`.

**maeehart** (2026-10-02T23:36:13Z):

/ci retry

**github-actions** (2026-10-02T23:36:31Z):

✅ Queued 1 failed job(s) for retry in [Buildkite CI #92699](https://buildkite.com/vllm/ci/builds/92699).

