---
title: '59591'
pageLayout: default
slug: thoughts/inference/vllm-prs/59591
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59591.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59591](https://github.com/aarnphm/aarnphm.github.io/issues/59591) \[ROCm\]\[Perf\] Kimi-K3 Store only the current rank’s shards in latent MoE up-proj

[vllm-project/vllm#59591](https://github.com/vllm-project/vllm/pull/59591) · kliuae · MERGED · created 2026-10-01T10:51:08Z · merged 2026-10-08T14:55:21Z

## Purpose

On the ROCm path, the routed up-projection in Kimi-K3’s latent MoE is built as a `ReplicatedLinear` where each rank holds the full weight. However, the fused tail’s `ROCmLatentMoERunner` doesn’t need the full weight. Instead, each rank consumes its own shard of up-projection weights, in a column parallel manner, and then a later all-reduce puts the slices back together.

At TP=8, this means that 7/8 of the weights, despite resident on the HBM, is never touched in the fused latent tail path. Across Kimi-K3’s 92 MoE layers, this constitute to \~3.9GB per rank of HBM consumed by unused weights, and takes spaces away from useful pools like KV cache.

This PR addresses this by storing only the shard that each rank needs for the latent MoE up-projection, using `ColumnParallelLinear` instead. The computation stays the same as the old fused latent path was already doing the same weight-shard extraction in the live path, and ColumnParallelLinear’s weight loader loads only the per-rank shard that it computes on.

In agentic workloads, the saved space from model weights increases the prefix cache pool and hit rate, and improves throughput by \~0.9-30% across concurrencies when KV cache constitutes a binding constraint.

## Test Plan

lm_eval with gsm8k on MI355X x8

## Test Result

<details>
<summary>serving command</summary>



```
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE_SITUV2=1
export VLLM_ROCM_AITER_MLA_ASM_PADDING=asm
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export VLLM_SERVER_DEV_MODE=1
export AITER_BF16_FP8_MOE_BOUND=0
export AITER_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K3 --trust-remote-code --moe-backend auto \
  --tensor-parallel-size 8 --gpu-memory-utilization 0.9 --language-model-only \
  --max-num-seqs $((2*CONC)) --max-model-len 1048576 --enable-prefix-caching \
  --kv-cache-dtype fp8 --max-num-batched-tokens 8192 \
  --attention-config '{"mla_prefill_backend":"ROCM_AITER_FA"}' \
  --attention-backend ROCM_AITER_MLA --decode-context-parallel-size 8 --dcp-comm-backend a2a \
  --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 \
  --compilation-config '{"mode":3,"cudagraph_mode":"FULL","custom_ops":["+fused_rms_norm_gated"]}'
```

</details>

lm_eval gsm8k

<details>
<summary>eval command</summary>



```
lm_eval --model local-chat-completions \
  --model_args "model=moonshotai/Kimi-K3,base_url=http://localhost:8000/v1/chat/completions,\
num_concurrent=64,max_retries=3,tokenized_requests=False,max_gen_toks=8192,timeout=3600" \
  --tasks gsm8k --num_fewshot 20 --apply_chat_template --fewshot_as_multiturn --seed 1234
```

</details>

main

| Tasks | Version | Filter           | n-shot | Metric      |   |  Value |   | Stderr |
| ----- | ------: | ---------------- | -----: | ----------- | - | -----: | - | -----: |
| gsm8k |       3 | flexible-extract |     20 | exact_match | ↑ | 0.9712 | ± | 0.0046 |
|       |         | strict-match     |     20 | exact_match | ↑ | 0.9697 | ± | 0.0047 |

PR

| Tasks | Version | Filter           | n-shot | Metric      |   |  Value |   | Stderr |
| ----- | ------: | ---------------- | -----: | ----------- | - | -----: | - | -----: |
| gsm8k |       3 | flexible-extract |     20 | exact_match | ↑ | 0.9704 | ± | 0.0047 |
|       |         | strict-match     |     20 | exact_match | ↑ | 0.9704 | ± | 0.0047 |

AgentX benchmark runs were performed to reflect agentic workloads with prefix caching use

<details>
<summary>benchmark command</summary>



```
aiperf profile --scenario inferencex-agentx-mvp \
  --url http://127.0.0.1:8000 --endpoint /v1/chat/completions --endpoint-type chat --streaming \
  --model moonshotai/Kimi-K3 --tokenizer <k3-tokenizer> \
  --public-dataset semianalysis_cc_traces_weka_062126 --max-context-length 1048576 \
  --num-dataset-entries 393 --concurrency $CONC --benchmark-duration 1800 --random-seed 42 \
  --trajectory-start-min-ratio 0.25 --trajectory-start-max-ratio 0.75 \
  --trace-idle-gap-cap-seconds 300
```

</details>

| Conc | Branch | Prefix Cache Hit Rate (%) | Delta (pp) | Request Throughput (requests/hour) | Delta (%) | TTFT (ms) p50 | Delta (%) | ITL (ms) p50 | Delta (%) | Total Token Throughput (tokens/sec) | Delta (%) | Output Token Throughput (tokens/sec) | Delta (%) |
| ---- | ------ | ------------------------- | ---------- | ---------------------------------- | --------- | ------------- | --------- | ------------ | --------- | ----------------------------------- | --------- | ------------------------------------ | --------- |
| 32   | main   | 88.7                      | —          | 1950.7                             | —         | 933.6         | —         | 40.82        | —         | 61050.2                             | —         | 424.99                               | —         |
| 32   | PR     | 89.1                      | +0.3       | 1968.3                             | +0.90     | 953.2         | +2.10     | 39.58        | -3.04     | 61501.6                             | +0.74     | 429.84                               | +1.14     |
| 48   | main   | 87.4                      | —          | 2013.3                             | —         | 1177.3        | —         | 53.08        | —         | 75145.5                             | —         | 427.18                               | —         |
| 48   | PR     | 87.6                      | +0.2       | 2038.7                             | +1.26     | 1172.8        | -0.38     | 52.89        | -0.36     | 75715.5                             | +0.76     | 440.49                               | +3.12     |
| 64   | main   | 81.1                      | —          | 1719.8                             | —         | 1677.1        | —         | 87.73        | —         | 67222.4                             | —         | 350.73                               | —         |
| 64   | PR     | 83.3                      | +2.2       | 1868.5                             | +8.65     | 1561.1        | -6.92     | 77.14        | -12.06    | 72562.1                             | +7.94     | 379.13                               | +8.10     |
| 70   | main   | 74.8                      | —          | 1273.7                             | —         | 2406.9        | —         | 102.36       | —         | 50756.1                             | —         | 227.39                               | —         |
| 70   | PR     | 80.1                      | +5.3       | 1668.9                             | +31.03    | 2029.9        | -15.66    | 94.76        | -7.42     | 65427.9                             | +28.91    | 325.35                               | +43.08    |

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**tjtanaa** (2026-10-02T05:50:55Z):

/ci run

**github-actions** (2026-10-02T05:51:14Z):

❌ This PR is 211 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5946352434 -->

**tjtanaa** (2026-10-07T14:32:59Z):

/ci run

**github-actions** (2026-10-07T14:35:02Z):

✅ Triggered [Buildkite CI #93331](https://buildkite.com/vllm/ci/builds/93331) for commit `3cf62a07f037`.

**vllm-agent** (2026-10-07T15:05:43Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 2 test steps (2 jobs) instead of 36 (52 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 36 (52)       | 2 (2)    | 35 (51)    | 1 (1)     |
| AMD mirrors            | 33 (43)       | 8 (24)   | 30 (37)    | 5 (18)    |

<details><summary>Selector would run (2)</summary>

- `cpu-spec-decode-tests`
- `kimi-k3-unit-tests-b200`

</details>

<details><summary>Would skip (today's rules run them) (35)</summary>

- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `cpu-language-generation-and-pooling-model-tests` ×3
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `kimi-k3-prefix-cache-4xb200` ×2
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `rl-entrypoints-tests`

</details>

<details><summary>Would add (today's rules do not run them) (1)</summary>

- `cpu-spec-decode-tests` (code map)

</details>

<details><summary>AMD mirrors: would skip (30)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`

</details>

<details><summary>AMD mirrors: would add (5)</summary>

- `basic-models-tests-extra-initialization` ×14 (code map)
- `batch-invariance-h100` (code map)
- `pytorch-nightly-dependency-override-check` (code map)
- `ray-dependency-compatibility-check` (code map)
- `torch-stable-abi-audit` (code map)

</details>

<sub>3 changed files · base `c83935b300` · head `3cf62a07f0` · Python record: build 93039 at `1e5d0ea888` · kernel record: table [`43b4aae`](https://github.com/aarnphm/aarnphm.github.io/commit/43b4aaea3e) (build 93244), map [`43b4aae`](https://github.com/aarnphm/aarnphm.github.io/commit/43b4aaea3e) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 9 optional steps the selector would also run</sub>

