---
title: '51725'
pageLayout: default
slug: thoughts/inference/vllm-prs/51725
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/51725.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#51725](https://github.com/aarnphm/aarnphm.github.io/issues/51725) \[Perf\] Adaptive budget for spec scheduled token, 55%\~65% E2E TTFT Improvement

[vllm-project/vllm#51725](https://github.com/vllm-project/vllm/pull/51725) · yewentao256 · MERGED · created 2026-08-10T19:34:49Z · merged 2026-08-11T03:52:22Z

## Purpose

Let’s consider `max_num_seqs=1024`and `max_num_batched_tokens=8192` (by default)

| Actual Request num | Old logic scheduled tokens | Now  |
| ------------------ | -------------------------- | ---- |
| 1                  | 2048                       | 8186 |
| 32                 | 2048                       | 8000 |
| 128                | 2048                       | 7424 |
| 1024               | 2048                       | 2048 |

This PR make the scheduled tokens much larger when request num is small by using an adaptive strategy

This is **extremely helpful** when request num is small and give very huge perf improvement

## Test

Acc covered in unit tests



```bash
vllm serve moonshotai/Kimi-K3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.95 \
  --load-format fastsafetensors \
  --enable-prefix-caching \
  --reasoning-parser kimi_k3 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 30000 \
  --speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

 HF_HUB_CACHE=/data/engine/hub_cache HF_HUB_OFFLINE=1 guidellm run   --backend '{"kind":"openai_http","target":"http://127.0.0.1:30000","model":"moonshotai/Kimi-K3","request_format":"/v1/chat/completions","timeout":100000,"extras":{"body":{"reasoning_effort":"max","temperature":1.0,"top_p":0.95}}}'   --tokenizer '{"kind":"huggingface_auto","model":"moonshotai/Kimi-K3","load_kwargs":{"trust_remote_code":true,"local_files_only":true}}'   --data 'kind=synthetic_text,prompt_tokens=8000,output_tokens=1000'   --profile 'kind=concurrent,warmup=0.1,cooldown=0.1'   --override profile.streams '1,4,16'   --constraint 'kind=max_duration,seconds=150'   --constraint 'kind=max_errors,count=10'   --seed 'kind=static,value=42'   --metrics 'kind=generative,sample_size=20'   --output 'kind=json,path=/home/yewentao256/guidellm-results/output_vllm_kimik3_8k1k.json'
```

And we can get (benchmark result combined [#51726](https://github.com/aarnphm/aarnphm.github.io/issues/51726), I make it separate so easier to reveiw / revert if issues bump out

| Request num | throughput | e2e lantency              | TTFT                      |
| ----------- | ---------- | ------------------------- | ------------------------- |
| 1           | +6.5%🚀    | 7.09s → 6.31s（-11.0%）🚀   | 1417ms → 491ms（-65.3%）🚀  |
| 4           | +14.7%🚀   | 14.55s → 11.80s（-18.9%）🚀 | 1369ms → 609ms（-55.5%）🚀  |
| 16          | +41.5%🚀   | 32.29s → 24.81s（-23.1%）🚀 | 3218ms → 1217ms（-62.2%）🚀 |

[output_vllm_kimik3_8k1k.json](https://github.com/user-attachments/files/30911148/output_vllm_kimik3_8k1k.json)
[output_vllm_kimik3_8k1k_main.json](https://github.com/user-attachments/files/30911149/output_vllm_kimik3_8k1k_main.json)

---

## comments

**yewentao256** (2026-08-10T19:56:03Z):

/ci run

**github-actions** (2026-08-10T19:56:23Z):

✅ Triggered [Buildkite CI #83199](https://buildkite.com/vllm/ci/builds/83199) for commit `8d750a90c1a8`.

**yewentao256** (2026-08-10T20:43:01Z):

/ci run

**github-actions** (2026-08-10T20:43:24Z):

✅ Triggered [Buildkite CI #83207](https://buildkite.com/vllm/ci/builds/83207) for commit `e4f456acbea5`.

**njhill** (2026-08-11T01:50:43Z):

/ci run

**github-actions** (2026-08-11T01:51:00Z):

✅ Triggered [Buildkite CI #83264](https://buildkite.com/vllm/ci/builds/83264) for commit `29a8cba9f23f`.

