---
title: '56159'
pageLayout: default
slug: thoughts/inference/vllm-prs/56159
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56159.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56159](https://github.com/aarnphm/aarnphm.github.io/issues/56159) \[Kimi K3 Perf\] Avoid KDA mixed-batch gather/scatter, 5.2%\~7.7% E2E Throughput Improvement

[vllm-project/vllm#56159](https://github.com/vllm-project/vllm/pull/56159) · yewentao256 · MERGED · created 2026-09-09T21:51:01Z · merged 2026-09-10T09:57:01Z

## Purpose

Part of [vllm-project/vllm#50587](https://github.com/vllm-project/vllm/issues/50587)



```bash
# Example packed batch:
Token order: [N0, N1, S0, S1, S2]
              └─ N ─┘  └──── S ────┘

Packed QKV / gate / beta
          |
          +-- index_select(non-spec) × 3 --> non-spec KDA --+
          |                                                  |
          +-- index_select(spec)     × 3 --> spec KDA -------+
                                                             |
                              index_copy_(non-spec output) ---+
                              index_copy_(spec output) -------+
                                                             |
```

Now



```bash
Packed QKV / gate / beta: [N0, N1 | S0, S1, S2]
                                  |
                +-----------------+-----------------+
                |                                   |
       zero-copy slice [0:2]              zero-copy slice [2:5]
                |                                   |
        non-spec KDA                         spec KDA
     out=final_output[0:2]              out=final_output[2:5]
                |                                   |
                +-----------------+-----------------+
                                  |
                    Final output already assembled
```

So we save 6 `index_select` and 2 `index_copy_` for each layer

## Test



```bash

vllm serve moonshotai/Kimi-K3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --load-format fastsafetensors \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching \
  --reasoning-parser kimi_k3 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 30000 \
  --max-model-len auto \
  --max-num-seqs 48 \
  --use-replayssm \
  --speculative-config '{"model":"RedHatAI/Kimi-K3-speculator.dspark","method":"dspark","num_speculative_tokens":8,"draft_sample_method":"probabilistic","rejection_sample_method":"standard"}'

```

### Acc



```bash

lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:30000/v1/completions,model=$MODEL,num_concurrent=1024" --tasks gsm8k --trust_remote_code

Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9636|±  |0.0052|
|     |       |strict-match    |     5|exact_match|↑  |0.9636|±  |0.0052|
```

### Perf



```bash
HF_HUB_CACHE=/data/engine/hub_cache HF_HUB_OFFLINE=1 guidellm run   --backend '{"kind":"openai_http","target":"http://127.0.0.1:30000","model":"moonshotai/Kimi-K3","request_format":"/v1/chat/completions","timeout":100000,"extras":{"body":{"reasoning_effort":"max","temperature":1.0,"top_p":0.95}}}'   --tokenizer '{"kind":"huggingface_auto","model":"moonshotai/Kimi-K3","load_kwargs":{"trust_remote_code":true,"local_files_only":true}}'   --data 'kind=synthetic_text,prompt_tokens=8000,output_tokens=1000'   --profile 'kind=concurrent,warmup=0.1,cooldown=0.1'   --override profile.streams '1,4,16'   --constraint 'kind=max_duration,seconds=150'   --constraint 'kind=max_errors,count=10'   --seed 'kind=static,value=42'   --metrics 'kind=generative,sample_size=20'   --output 'kind=json,path=/home/yewentao256/guidellm-results/output_vllm_kimik3_8k1k.json'

```

| Concurrency | Main Avg Latency (Base) | Current Avg Latency | Main Throughput (Base) | Current Throughput     | Main Avg TTFT (Base) | Current Avg TTFT    |
| ----------- | ----------------------- | ------------------- | ---------------------- | ---------------------- | -------------------- | ------------------- |
| 4           | 11.29 s                 | 10.63 s (−5.8%) 🚀  | 352.7 tok/s            | 379.8 tok/s (+7.7%) 🚀 | 448.3 ms             | 440.4 ms (−1.8%) 🚀 |
| 16          | 21.62 s                 | 21.16 s (−2.1%) 🚀  | 723.7 tok/s            | 761.3 tok/s (+5.2%) 🚀 | 930.9 ms             | 898.3 ms (−3.5%) 🚀 |

For batch size 1 we nearly keep the same as expected

| Concurrency | Main Avg Latency (Base) | Current Avg Latency | Main Throughput (Base) | Current Throughput    | Main Avg TTFT (Base) | Current Avg TTFT   |
| ----------- | ----------------------- | ------------------- | ---------------------- | --------------------- | -------------------- | ------------------ |
| 1           | 6.21 s                  | 6.20 s (−0.2%) ➖    | 165.6 tok/s            | 165.9 tok/s (+0.2%) ➖ | 379.0 ms             | 380.7 ms (+0.4%) ➖ |

---

## comments

**yewentao256** (2026-09-09T21:51:18Z):

/ci run

**github-actions** (2026-09-09T21:51:35Z):

✅ Triggered [Buildkite CI #87998](https://buildkite.com/vllm/ci/builds/87998) for commit `c42435ed19e0`.

