---
title: '54015'
pageLayout: default
slug: thoughts/inference/vllm-prs/54015
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/54015.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#54015](https://github.com/aarnphm/aarnphm.github.io/issues/54015) \[Kimi-K3\] Merge MLA gate into QKV-A projection

[vllm-project/vllm#54015](https://github.com/vllm-project/vllm/pull/54015) · gau-nernst · MERGED · created 2026-08-27T07:44:15Z · merged 2026-08-28T00:47:19Z

## Purpose

This PR merges MLA’s `g_proj` (used for output gating) into QKV_a projection to create a larger GEMM. We don’t expect this to have a large impact as the original QKV_a gemm is already quite large. The main purpose of this PR is to prepare for the upcoming fused AG-GEMM (used in SP), which benefits significantly from having a larger GEMM.

What makes this PR a bit more complicated is that currently we have a low-latency optimization for MLA: `g_proj` runs in a side stream when bs<512, so that `g_proj` can overlap with `qkv_a_proj`+MLA. From my testing, this side-stream optimization is faster than running a combined `qkv_a_g_proj` GEMM at bs=1, hence we should still keep this optimization.

To do so, this PR does **weight slicing** when `bs<512` to run `g_proj` in side stream, preserving the current behavior. When `bs>=512`, the merged GEMM replaces the sequential GEMM baseline.

Dispatch logic

- Entrypoint: `forward()`
  - Q full rank (Kimi-Linear): `_forward_full_rank_q`. This is unchanged, because Kimi-Linear doesn’t use output gate
  - Q low-rank (Kimi-K3): `_forward_q_lora`
    - When running g_proj in side stream is eligible (M<512, BF16 weights): slice weight and run (qkv_a+MLA) || (g)
    - Otherwise, run qkv_a_g together

Limitations:

- Since we do weight slicing here, we must handle each quantization explicitly (i.e. for quantized weights, need to slice the scale factors as well. Also need to manually dispatch the GEMM function, can’t rely on Linear module).
- This PR currently only supports BF16 weight slicing (official Kimi-K3 weights). In other words, if QKV_a and g_proj are **not BF16, running `g_proj` in side stream is disabled**. This may LL perf of 3rd party quantizations, such as `nvidia/Kimi-K3-NVFP4`. We can expand support as needed cc @wzhao18
- `bs<512` threshold might not be optimal. This PR does not re-tune this threshold.

## Test Plan

TP8 on GB300

- GSM8K: 0.9636
- OCRBench: 89.4

## E2E perf results

TP8 on GB300



```
VLLM_ALLREDUCE_USE_FLASHINFER=1
VLLM_USE_RUST_FRONTEND=1

vllm serve {model}
--port {port}
-tp 8
--nnodes {nnodes}
--node-rank {node_rank}
--master-addr {master_addr}
--trust-remote-code
--load-format fastsafetensors
--no-enable-prefix-caching
--kv-cache-dtype fp8
--attention-config '{"mla_prefill_backend":"flashinfer","use_prefill_query_quantization":true}'
--reasoning-parser kimi_k3
--tool-call-parser kimi_k3
--enable-auto-tool-choice
```



```
VLLM_USE_RUST_BENCH=1

vllm bench serve
--backend openai-chat
--base-url http://{head_addr}:8000
--model {model}
--dataset-name speed-bench
--speed-bench-config throughput_8k
--speed-bench-max-input-len 8192
--output-len 1024
--ignore-eos
--num-warmups 5
--sweep-max-concurrency 1,4,16
--sweep-num-prompts-factor 10
--percentile-metrics "ttft,tpot,itl,e2el"
--metric-percentiles 50,90,99
--result-dir "{log_dir}/results"
--save-result
```

Baseline commit: [`f25c580`](https://github.com/aarnphm/aarnphm.github.io/commit/f25c580af15986f7ccdaaf7639c15896e04cf384)

TTFT

| Concurrency | P50 (before) | P50 (after)         | P90 (before) | P90 (after)         |
| ----------- | ------------ | ------------------- | ------------ | ------------------- |
| 1           | 342.27 ms    | 342.08 ms (-0.06%)  | 344.98 ms    | 345.37 ms (+0.11%)  |
| 4           | 838.35 ms    | 834.83 ms (-0.42%)  | 1006.45 ms   | 1003.79 ms (-0.26%) |
| 16          | 1535.16 ms   | 1525.42 ms (-0.63%) | 2392.26 ms   | 2377.69 ms (-0.61%) |

Interactivity (1/TPOT)

| Concurrency | P50 (before) | P50 (after)           | P90 (before) | P90 (after)           |
| ----------- | ------------ | --------------------- | ------------ | --------------------- |
| 1           | 116.14 tok/s | 115.87 tok/s (-0.23%) | 116.14 tok/s | 115.87 tok/s (-0.23%) |
| 4           | 85.25 tok/s  | 85.18 tok/s (-0.08%)  | 84.39 tok/s  | 84.32 tok/s (-0.08%)  |
| 16          | 56.24 tok/s  | 56.21 tok/s (-0.05%)  | 54.64 tok/s  | 55.62 tok/s (+1.79%)  |

Small improvement for TTFT (prefill). Decode is mostly noise, except improvement in P90 TPOT at concurrency=16

## Test Result

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**gau-nernst** (2026-08-27T11:59:18Z):

/ci run

**github-actions** (2026-08-27T11:59:37Z):

✅ Triggered [Buildkite CI #85807](https://buildkite.com/vllm/ci/builds/85807) for commit `bc5033efed4f`.

