---
title: '58201'
pageLayout: default
slug: thoughts/inference/vllm-prs/58201
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/58201.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#58201](https://github.com/aarnphm/aarnphm.github.io/issues/58201) \[ROCm\]\[Kimi-K3\] Make VLLM_ROCM_USE_AITER_MOE_SITUV2 select a4w4/a8w4/a16w4

[vllm-project/vllm#58201](https://github.com/vllm-project/vllm/pull/58201) · hongxiayang · MERGED · created 2026-09-22T19:48:36Z

## Purpose

The SiTU MXFP4 MoE has three FlyDSL kernel families that differ only in activation precision; the vLLM
flag used to be a bool (on = a4w4, off = a16w4) with no way to pick a8w4. Now it takes the dtype name so
the three can be A/B’d on the same server config.

K3 MoE activation option — `VLLM_ROCM_USE_AITER_MOE_SITUV2=a16w4|a8w4|a4w4`

| value                                  | AITER env vLLM exports | stage-1 kernel family      | tuned CSV (aiter)             |
| -------------------------------------- | ---------------------- | -------------------------- | ----------------------------- |
| `a16w4` (also `0`)                     | neither                | `flydsl_moe1_abf16_wfp4_*` | `kimik3_a16w4_tuned_fmoe.csv` |
| `a8w4`                                 | `AITER_SITUV2_A8W4=1`  | `flydsl_moe1_afp8_wfp4_*`  | `kimik3_a8w4_tuned_fmoe.csv`  |
| `a4w4` (**default**: unset/`auto`/`1`) | `AITER_SITUV2_A4W4=1`  | `flydsl_moe1_afp4_wfp4_*`  | `kimik3_a4w4_tuned_fmoe.csv`  |

Unknown values raise `ValueError` at env read. The old `VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4` alias is no longer read (it now has no effect; default is a4w4 anyway).

### Serve command



```bash
# 8× MI350X (gfx950), TP8. Only VLLM_ROCM_USE_AITER_MOE_SITUV2 changes between arms.
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE_SITUV2=a4w4        # a4w4 (default) | a8w4 | a16w4
export VLLM_ROCM_AITER_MLA_ASM_PADDING=asm
export VLLM_USE_BREAKABLE_CUDAGRAPH=1            # K3 is not torch.compile-able on this nightly
export SAFETENSORS_FAST_GPU=1

vllm serve /data/Kimi-K3-public --trust-remote-code \
  --served-model-name moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 --mm-encoder-tp-mode data \
  --gpu-memory-utilization 0.95 --max-num-seqs 64 \
  --max-model-len 10240 --max-num-batched-tokens 4096 \
  --reasoning-parser kimi_k3
```

### niah_single_2 A/B (2026-09-22, K3 TP8, `--max-model-len 40960 --max-num-batched-tokens 8192`, 500 samples/length, greedy, raw completions)



```
lm_eval --model local-completions \
  --model_args "model=moonshotai/Kimi-K3,base_url=http://127.0.0.1:8005/v1/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer=/data/Kimi-K3-public,trust_remote_code=True,max_length=131072,timeout=3600" \
  --tasks niah_single_2 --metadata '{"max_seq_lengths":[4096,8192,16384,32768]}' --gen_kwargs temperature=0
```

| arm            |    4k |    8k |   16k |   32k |
| -------------- | ----: | ----: | ----: | ----: |
| a16w4          | 0.998 | 0.994 | 0.990 | 0.996 |
| a8w4           | 0.996 | 0.996 | 0.996 | 0.996 |
| a4w4 (default) | 0.996 | 1.000 | 0.998 | 0.998 |

All three ≥ 0.99 at every length; differences are within 1–5 samples of 500 (noise). The MoE activation
precision is **not** what breaks niah_single_2 on this stack (vLLM dev580 + AITER v0.1.21.post1, no spec decode,
prefix caching on). The 2.1%-err a4w4 row at token=8192 was exercised (8192-token prefill chunks) with no visible
effect.

### 8k/1k perf + gsm8k per option (2026-09-22, one serve session per arm: bench conc 1/32/64 → gsm8k full 1319)

Serve config = the recipe command above (`--max-model-len 10240 --max-num-batched-tokens 4096`, prefix caching on).
Random 8k ISL / 1k OSL, ignore-eos, num_prompts = max(20, 10×conc). gsm8k = canonical 20-shot chat harness.



```bash
# gsm8k — canonical K3 harness (20-shot chat, fewshot-as-multiturn, 2048 gen tokens, seed 1234, full 1319)
lm_eval --model local-chat-completions \
  --model_args "model=moonshotai/Kimi-K3,base_url=http://127.0.0.1:8005/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,max_gen_toks=2048,timeout=3600" \
  --tasks gsm8k --num_fewshot 20 --apply_chat_template --fewshot_as_multiturn --seed 1234
# 8k/1k perf — per concurrency C, num_prompts = max(20, 10*C)
vllm bench serve --backend openai --host 127.0.0.1 --port 8005 --endpoint /v1/completions \
  --model moonshotai/Kimi-K3 --trust-remote-code --dataset-name random \
  --random-input-len 8192 --random-output-len 1024 --random-range-ratio 0.0 \
  --num-prompts $((C*10)) --max-concurrency $C --request-rate inf --ignore-eos --temperature 0
```

| arm            | conc | tok/s | mean TTFT ms | mean TPOT ms |  gsm8k flexible |    gsm8k strict |
| -------------- | ---: | ----: | -----------: | -----------: | --------------: | --------------: |
| a16w4          |    1 |  51.5 |          716 |        18.75 | 0.9788 ± 0.0040 | 0.9795 ± 0.0039 |
| a16w4          |   32 | 659.7 |         2456 |        46.06 |                 |                 |
| a16w4          |   64 | 838.8 |         3715 |        72.55 |                 |                 |
| a8w4           |    1 |  52.2 |          693 |        18.49 |  0.975 ± 0.0043 |  0.975 ± 0.0043 |
| a8w4           |   32 | 688.8 |         2340 |        44.12 |                 |                 |
| a8w4           |   64 | 875.6 |         3518 |        69.52 |                 |                 |
| a4w4 (default) |    1 |  51.9 |          673 |        18.62 | 0.9644 ± 0.0051 | 0.9644 ± 0.0051 |
| a4w4 (default) |   32 | 697.8 |         2262 |        43.60 |                 |                 |
| a4w4 (default) |   64 | 895.4 |         3383 |        68.04 |                 |                 |

## Test Plan

## Test Result

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**hongxiayang** (2026-09-22T21:48:18Z):

cc @tjtanaa to help review

**tjtanaa** (2026-09-26T10:42:36Z):

/ci run

**github-actions** (2026-09-26T10:42:55Z):

✅ Triggered [Buildkite CI #91383](https://buildkite.com/vllm/ci/builds/91383) for commit `6138175be867`.

