---
title: '61024'
pageLayout: default
slug: thoughts/inference/vllm-prs/61024
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/61024.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#61024](https://github.com/aarnphm/aarnphm.github.io/issues/61024) \[Roadmap\] vLLM Roadmap Q4 2026

[vllm-project/vllm#61024](https://github.com/vllm-project/vllm/issues/61024) · simon-mo · OPEN · created 2026-10-10T19:21:47Z

In [#32455](https://github.com/vllm-project/vllm/issues/32455), we broke down vLLM’s goals into various special interest groups (SIGs). Building on the [Q3 roadmap](https://github.com/vllm-project/vllm/issues/48168), please find below each SIG’s Q4 roadmap. You can find regular meetings of these SIGs on the public vLLM calendar.

## Key takeaway

- **Agentic Workload:** Q3 established Model Runner V2 as the default and brought production-ready Rust frontend and distributed KV offloading support. Q4 focuses on making optimized agentic serving easier to deploy: better day-0 performance, tuned production recipes, adaptive speculative decoding, richer agent hints, and fault tolerance across hardware.
- **Ecosystem:** We are expanding low-precision serving and checkpoint conversion across NVIDIA and AMD, strengthening speculative decoding and RL integration, and advancing Agentic API and vLLM-Omni. Frontier models and new hardware need a clear path from initial support to validated, optimized production configurations.
- **OSS Maintenance:** We are simplifying engine and model interfaces, completing quantization dispatch refactors, and improving CI coverage, time to signal, security, and performance evaluation. Agent-assisted review and better test-quality checks should help the community maintain a growing model, feature, and hardware matrix.

### SIG Core ([Meeting Notes](https://docs.google.com/document/d/1x36yhn655LiuV6bPK2C1M-6jfMzgXDLQf2PIngJndgc/edit?tab=t.0))

- [ ] Fully remove Model Runner V1 and close the remaining [Model Runner V2 gaps](https://github.com/vllm-project/vllm/issues/47172).
- [x] Implement [extensible KV cache support](https://github.com/vllm-project/vllm/pull/56492), pluggable `KVCacheConfigBuilder` implementations for models, and better cache-grouping heuristics for sliding-window attention speculators.
- [ ] Refactor CUDA graphs: separate prefill/decode graph sizes; clarify capture, dispatch, and padding semantics, including adaptive verification and sequence parallelism; and simplify encoder and DeepSeek V4.1 prefill-skipping graphs. Validate prefill graph sizing against padding overhead and performance regressions.
- [ ] Simplify and harden the engine’s model loop, data-parallel flow, and scheduler, and improve streaming-input reliability.
- [ ] Reduce coupling between model implementations and the engine through a clearer model interface.
- [ ] Advance the Rust frontend toward default use, including resolving Responses API readiness and parity requirements.
- [ ] Add native per-step observability through [forward-pass metrics](https://github.com/vllm-project/vllm/pull/52061), in collaboration with Dynamo.

* Aspirational work:
  - Simplify KV connector interfaces through standardized KV layouts, including exploring parallel interfaces.
  - Revisit automatic prefix caching and investigate lookahead EAGLE block caching as a replacement for block dropping.
  - Explore a more comprehensive scheduler redesign beyond the immediate cleanup.

### SIG Large Scale Serving ([Meeting Notes](https://docs.google.com/document/d/15A9zid75c41pcn8hzIKXOW2R6VFgeUfGuIMq-ebgEEI/edit?tab=t.0#heading=h.prqroxfe0fr1))

- [ ] Extend agent hints across routers and KV connectors with Dynamo and llm-d: multi-tenancy, cache-tier TTLs, filesystem eviction, and request-level control of prefetching, eviction, and selective offloading.
- [ ] Publish production recipes with clear explanations of deployment and performance tradeoffs, including disaggregated prefill/decode and KV offloading.
- [ ] Harden numerical fault handling, including NaNs, and expert-parallel recovery through restart and scale-down; integrate recovery with llm-d and Dynamo.
- [ ] Achieve state-of-the-art agentic workload performance for priority models across AMD and new hardware, including Rubin and MI455X.
- [ ] Integrate native [forward-pass metrics](https://github.com/vllm-project/vllm/pull/52061) into the serving observability work with Dynamo.

* Proposed work: define the scope and milestones for AFD.

### SIG Model Performance ([Meeting Notes](https://docs.google.com/document/d/1A1TOuCTPQ_72npOSzhWKfYOmDIF7OoagLGaL8gHe1TQ/edit?tab=t.0#heading=h.wmm9txlfqo9w))

- [ ] Improve day-0 performance and the workflow from initial model support to optimized agentic serving configurations.
- [ ] Continue sprints for frontier models, including DeepSeek V4.1, GLM 5.3, and Qwen3.8, with continuous model and feature tracking.
- [ ] Optimize Vera Rubin performance and publish reproducible serving recipes.
- [ ] Expand FP4 KV cache support to more models and add disk-backed KV offloading.
- [ ] Continue the [performance-evaluation dashboard](https://vllm-ci-dashboard.vercel.app/perf) and broaden regression tracking with SIG CI.

* Proposals to evaluate: bring `vllm-recipes` into the main repository, restructure model-specific tests, and clarify the scope and feasibility of deprecating vLLM IR and custom Torch compile passes.

### SIG Spec Decode ([Meeting Notes](https://docs.google.com/document/d/1DlgW4dWZyGup_BOZm24Bo50AFIiQgPXfSLqMGDLt6Ic/edit?tab=t.0#heading=h.1efi34fxtsr2))

- [ ] Improve HMA and heterogeneous attention support between drafter and verifier. Make sliding-window drafters compose efficiently with different target attention types and reduce KV cache padding and memory overhead.
- [ ] Polish adaptive verification and extend support to Mamba2, GDN, and QSA. Research oracle verification to quantify remaining performance opportunities and guide optimization.
- [ ] Migrate speculative model code to Flat Model, then optimize fusion and kernels for low-latency inference, including rejection sampling.
- [ ] Enable block rejection sampling and probabilistic verification by default.
- [ ] Finalize `open-spec-config` configuration details for popular draft algorithms and implement draft-model loading.
- [ ] Improve configuration ergonomics and robustness with tool calling, reasoning, and structured outputs. Enabling speculative decoding itself by default remains an aspiration.

### SIG CI ([Meeting Notes](https://docs.google.com/document/u/2/d/18VJOuEyAZmRVILWpOtYx5jC4iFZLwjHVw5TAdfJnjXQ/edit?tab=t.0))

- [ ] Collect test-coverage data and maintain a feature-combination matrix, including prefill/decode disaggregation.
- [ ] Bring time to signal toward 30 minutes through automatic test selection, pytest-level granularity, automatic sharding, and default job timeouts.
- [ ] Migrate remaining four-L4 jobs to H200 MIG, reduce Docker build/pull and environment-setup overhead, and retire obsolete jobs for old models.
- [ ] Improve infrastructure-failure detection, quarantine, and retries, and carry forward flaky-test management.
- [ ] Perform offensive security testing of vLLM and make CI safe for projects throughout the vLLM ecosystem.
- [ ] Expand performance and accuracy evaluation to disaggregated serving and TPU, including InferenceX-style benchmark sweeps and throughput/latency Pareto curves.
- [ ] Develop agent-assisted reviewers and CI shepherds, with checks for unit-test quality.
- [ ] Clarify and document CI trigger paths and coverage for RDNA, XPU, and other accelerators, including projects maintained outside the main repository, to reduce cross-backend breakage.

### SIG Quantization

- [ ] **LUT-B (W3A8) on Rubin:** finalize the NVIDIA format specification, including tile sizes, codebooks, and scales ([Model Optimizer #2204](https://github.com/NVIDIA/Model-Optimizer/issues/2204), [compressed-tensors #801](https://github.com/vllm-project/compressed-tensors/pull/801)); provide an [LLM Compressor calibration recipe](https://github.com/vllm-project/llm-compressor/pull/3018), validated checkpoints, and optimized dense/MoE kernels with MXFP8 activations.
- [ ] **Production FP4 and lower-bit KV caches:** publish accuracy evaluations and recommended configurations for priority models. Cover long-context interactive serving and high-concurrency data generation, and compose with hybrid attention, speculative drafters, NIXL disaggregation, and tiered offloading.
  - Expand NVFP4 coverage across FlashMLA/DSA sparse attention, FA4, Rubin kernels, and [SM8x–SM12x enablement](https://github.com/vllm-project/vllm/pull/46963).
  - Advance ROCm low-bit KV work, including [UltraQuant](https://github.com/vllm-project/vllm/pull/57057) and the [Octave KV proposal](https://github.com/vllm-project/vllm/issues/59773), with platform-specific validation rather than assuming one format fits every backend.
- [ ] **Checkpoint requantization:** complete the [online quantization from arbitrary precision roadmap](https://github.com/vllm-project/vllm/issues/59062) through a common online quantization API, including [requantization](https://github.com/vllm-project/vllm/pull/55684), [activation QuantKey overrides](https://github.com/vllm-project/vllm/pull/51808), [shared experts](https://github.com/vllm-project/vllm/pull/55686), and [MLA `kv_b_proj`](https://github.com/vllm-project/vllm/pull/55741). Initial conversions include MXFP8 to FP8 per-channel/block on Hopper and MI300, and FP8/NVFP4 to MXFP4 across vendors.
- [ ] **Unified dispatch and inspectability:** finish QuantKey/QuantConfig adoption in [Quark](https://github.com/vllm-project/vllm/pull/56099), [ModelOpt MoE](https://github.com/vllm-project/vllm/issues/55339), and compressed-tensors. Declare weight and activation QuantKeys across methods, complete the [MoE oracle and linear-kernel migration](https://github.com/vllm-project/vllm/issues/54959), extend QuantKey into `FusedMoEQuantDesc`, and log selected kernels/backends by layer type.
- [ ] **Kernel consolidation:** make [Humming the default over Marlin on Hopper](https://github.com/vllm-project/vllm/pull/56997), then extend to SM10x/SM12x. Consolidate autotuning configurations and benchmark overlapping CuTe-DSL, b12x, FlashInfer TRTLLM, Marlin, Qutlass, and Humming paths to choose per-architecture defaults and retire superseded paths.
- [ ] **Compression ecosystem:** deliver day-0 NVFP4/MXFP4 checkpoints for priority models, advance [MoNe](https://github.com/vllm-project/vllm/pull/50415) and [REAP expert pruning](https://github.com/vllm-project/llm-compressor/pull/2906), and make `convert_checkpoint` the general format-conversion entry point.
- [ ] **Quality validation:** add quantized-model accuracy CI, tensor-parallel parity checks for online quantization, and quantized weight-reload correctness checks with the RL ecosystem.

* Research directions: LUT-B attention support, FP4 with rotations through QuTLASS, HIGGS, and QAD. Continue evaluating [legacy format deprecation](https://github.com/vllm-project/vllm/issues/30136).

### SIG AMD

- [ ] Close frontier-model prefill, decode, speculative decoding, and long-context performance gaps on ROCm, prioritizing DeepSeek V4/V4.1, Kimi K3, GLM 5.x, and Qwen3.x.
- [ ] Productionize context parallelism for sparse, MLA, and hybrid-attention models, and harden prefill/decode disaggregation and KV transfer.
- [ ] Improve multi-tier KV offloading across GPU, host memory, SSD, and distributed storage, including Mooncake, MoRI/UMBP, and NIXL integrations.
- [ ] Expand AITER kernels and fused FP8/MXFP8/FP4/MXFP4 execution paths, with automatic backend selection and fewer AMD-specific configuration requirements.
- [ ] Strengthen AMD/Quark day-0 support: FP4 and FP8-PTPC for priority CDNA models, plus INT4 and other low-bit formats for priority RDNA models. Coordinate online requantization and flexible activation QuantKey overrides with SIG Quantization.
- [ ] Advance UltraQuant and MXFP4 KV-cache/attention support on ROCm, and support MTP, EAGLE, and DFlash for priority models, including quantized draft heads.
- [ ] Maintain MI355X as the primary performance platform while closing MI300X/MI325X gaps, expanding RDNA support where applicable, and improving AMD CI parity, performance regression coverage, and ROCm release readiness. Document the RDNA validation and CI trigger path with SIG CI.

### SIG Agentic API

- [ ] Deliver [multi-agent APIs](https://github.com/vllm-project/agentic-api/issues/298).
- [ ] Add a [code interpreter tool](https://github.com/vllm-project/agentic-api/issues/308).
- [ ] Use the [shared Rust frontend library](https://github.com/vllm-project/agentic-api/issues/359) for direct vLLM engine inference.
- [ ] Add a [file search tool](https://github.com/vllm-project/agentic-api/pull/278).

### vLLM-Omni

- [ ] Continue optimizing frontier diffusion models, including MiniMax H3.
- [ ] Integrate with RL frameworks, including verl with ve-omni as the training framework.
- [ ] Expand acceleration backends and framework integrations, including FastVideo.
- [ ] Complete the [full-duplex redesign](https://github.com/vllm-project/vllm-omni/issues/7181), carrying forward Q3 prototyping work.

### RL Ecosystem

- [ ] Execute the [Q4 vLLM × RL roadmap](https://github.com/vllm-project/vllm/issues/58990): weight updates, training–inference consistency, and feature enablement such as LoRA for RL.
- [ ] Improve day-0 RL support for new models.
- [ ] Refactor weight updates and advance the auxiliary connector design to address correctness at the architectural level, including quantized weight reloads with SIG Quantization.

---

Please leave feedback and comments, and work directly with the relevant SIG areas for deeper collaboration.

---

## comments

**github-actions** (2026-10-10T19:22:37Z):

CC @hongxiayang @tjtanaa @vllmellm @giuseppegrossi for ROCm-related issue

**AlejandroParedesLT** (2026-10-13T05:49:16Z):

Working on the SIG Spec Decode bullet “optimize fusion and kernels for low-latency inference, including rejection sampling”: [#61344](https://github.com/aarnphm/aarnphm.github.io/issues/61344) (draft, AI-assisted / human-reviewed).

First commit folds temperature scaling into the MRV2 rejection-sampling kernels (`vllm/v1/worker/gpu/spec_decode/rejection_sampler_utils.py`), per the `TODO(mgoin)` in `rejection_sampler.py` about the FP32 `processed_logits` buffer / chunking workaround. Groundwork only so far, default behavior unchanged for every existing caller — caller-side wiring to actually drop that buffer is next on the same PR.

Searched open issues/PRs first and didn’t find this roadmap slice claimed elsewhere; flagging here for visibility per the duplicate-work check in AGENTS.md.

**AlejandroParedesLT** (2026-10-16T08:52:28Z):

Update on the Flat Model migration slice: [#61357](https://github.com/aarnphm/aarnphm.github.io/issues/61357) (Medusa) is closed — Medusa was removed from vLLM entirely in [#61701](https://github.com/aarnphm/aarnphm.github.io/issues/61701) while that PR was in flight. Picked HY V3 instead ([#61987](https://github.com/aarnphm/aarnphm.github.io/issues/61987), same rationale as Medusa: zero custom kernels, zero dependency on any other model file), now open and ready for review.

**fxmarty-amd** (2026-10-16T18:46:48Z):

@simon-mo could we add to the roadmap a standardized integration of monokernels in vllm? Related to [vllm-project/vllm#60904](https://github.com/vllm-project/vllm/issues/60904) and as monokernels are cross-vendors. I hope we can adopt common APIs/abstractions.

**jiangLLM** (2026-10-18T06:29:08Z):

I opened [#62258](https://github.com/aarnphm/aarnphm.github.io/issues/62258) for the oracle-verification item. The benchmark uses saved target-model outputs as draft tokens and compares throughput with EAGLE3.

Here are the [results and setup](https://gist.github.com/jiangLLM/e95756b974f385846e4277b7ee68d6c8) for Qwen3-8B on one H100, with async scheduling off.

**ammarali32** (2026-10-26T05:44:04Z):

Hello @simon-mo
we would like to present
Lossless state-tree representation for weights and KV cache (\~12 bits per value, bit-exact BF16)

### Motivation.

LLM decoding is memory-bound: every new token re-reads all the weights and the whole KV cache. Today the way to read fewer bytes in vLLM is quantization, which changes the model’s numerics.

We have been working on **state trees** [2610.31931](https://arxiv.org/abs/2610.31931) \[@ali2026weightpathshareddiscrete\] , a representation in which every value is a path of shared discrete decisions. Its simplest instance shares one exponent node per 64 BF16 values: every weight and every cached key/value is stored in about 12 bits and decodes to its **exact BF16 bits**, with no calibration and no accuracy trade-off.

We implemented it as an out-of-tree vLLM plugin with fused CUDA kernels <https://github.com/ammarali32/StateTree-Representation>:

- **Weights:** a `LinearMethodBase` for all linear layers, with fused decode-and-multiply kernels for decode and medium batches, and decode + cuBLAS for large prefill.
- **KV cache:** a custom attention backend that stores each 16-token block as a state-tree page, with a decode-attention kernel that reads the compressed pages directly.

Qwen3-8B, one A100-40GB, vLLM 0.32, `gpu_memory_utilization` 0.85:

|                                            | BF16        | statetree           |
| ------------------------------------------ | ----------- | ------------------- |
| Weights in GPU memory                      | 15.3 GiB    | 12.8 GiB            |
| KV-cache capacity                          | 119K tokens | 178K tokens (1.49×) |
| Decode, batch 1                            | 75.6 tok/s  | 97.2 tok/s (1.29×)  |
| Time to the complete answer                | 1.00×       | 1.10–1.28× faster   |
| Decode attention vs paged FlashAttention-2 | 1.00×       | 1.21–2.46× faster   |

Weights and KV values are bit-identical to BF16; outputs match up to floating-point summation order, the same kind of difference as switching GEMM kernels. Prefix caching, chunked prefill, CUDA graphs and tensor/pipeline parallelism all work, and it composes with existing quantization methods (layers they claim stay quantized).

Known limitations: prefill of the weight GEMM runs at 0.87–1.03× of cuBLAS (time to first token 3–15% longer), and the KV backend currently requires BF16 and head size 128.

### Proposed Change.

We would like to upstream this and would value your guidance on the preferred shape:

1. **Weights:** a new method under `vllm/model_executor/layers/quantization` (reuses the existing plumbing), or a separate lossless “representation” option, since it does not change the model’s numerics.
2. **KV cache:** a new attention backend selected by a flag (for example `--kv-cache-dtype statetree`), or kept as a plugin backend.
3. **Kernels:** CUDA sources built with vLLM, or loaded from the `statetree` package.

### Feedback Period.

Open. Happy to follow your process.

If you think this will be within your interest, we would be glad to open a PR, add tests (bit-exact round trips, generation equality against BF16) and benchmarks, and maintain the kernels. Would you consider this feature?

**huzaifaasim017** (2026-10-27T12:34:46Z):

For the proposed state-tree KV backend, I would add a cache-lifecycle test before choosing the upstream interface. The reported bit-exact round trip is useful, but a 16-token compressed page is also written incrementally during decode. A request can fork through prefix sharing, append to a partially filled block, then roll back speculative tokens or be cancelled. Those transitions need to preserve the exact pre-fork KV values and avoid exposing a partially re-encoded page to another sequence.

A small deterministic test could compare the plugin backend with BF16 after each append/fork/rollback (including a block boundary and copy-on-write), then run the same trace with prefix caching and chunked prefill enabled. I would also report bytes allocated and decode latency for partially filled blocks, since the 1.49x capacity number for steady state may change when many short agent requests leave tail blocks half full. That evidence would help decide whether the new representation belongs behind a KV cache dtype or a more explicit backend capability.

**0xDexe** (2026-11-05T20:53:37Z):

Hi! I’d like to contribute a slice of the “migrate speculative model code to Flat Model” item. Is MiMo (`mimo.py` + `mimo_mtp.py`, `MiMoForCausalLM` / `MiMoMTPModel`) open? I don’t see a PR for it.

I plan to move both to `vllm/models/mimo/common/{model,mtp}.py` (no platform-specific code, so no nvidia/amd split, following [#61987](https://github.com/aarnphm/aarnphm.github.io/issues/61987)), repoint the registry entries, no logic change. I can validate on a real GPU by running `tests/v1/e2e/spec_decode/mtp/mimo/test_mtp.py` with `XiaomiMiMo/MiMo-7B-Base` before and after.

Both files still import from the legacy `qwen2.py` (`Qwen2Model`, `Qwen2DecoderLayer`). Is that OK for this slice, or should Qwen2 move first?

cc @AlejandroParedesLT to avoid overlapping with your next slice.

