---
title: '59665'
pageLayout: default
slug: thoughts/inference/vllm-prs/59665
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59665.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59665](https://github.com/aarnphm/aarnphm.github.io/issues/59665) \[RFC\]: Fast-Track Merging for Model Optimization PRs

[vllm-project/vllm#59665](https://github.com/vllm-project/vllm/issues/59665) · benchislett · OPEN · created 2026-10-01T18:53:37Z

# \[RFC\]: Fast-Track Merging for Model Optimization PRs

### Motivation.

Frontier coding models have become extremely proficient at inference model optimization and efficient GPU kernel development. The frequency of new vLLM PRs is rapidly increasing, and is outpacing reviewer bandwidth.
Powered by fully autonomous optimization loops (e.g. “autoresearch”), optimization development can be scaled horizontally across a wide range of models, hardware architectures, and deployment scenarios.

The potential speedups are extremely large, but careful steps must be taken to ensure that vLLM’s code quality, usability, and reliability is not sacrificed in the process — this is why we have a comprehensive review process. Striking a balance between contribution barriers and performance upside is critical.

The fundamental goal of the fast-track is to merge PRs faster, with the same standard of quality. The way to achieve this is by systematically verifying aspects that ordinarily require manual reviewer verification (correctness, performance, no risk of breaking other workloads, etc.). By focusing on a subset of PRs whose changes are strictly contained to an isolated module (namely, the “model forward pass”) we can tailor guardrails to decrease reviewer involvement while maintaining a high bar for stability.

### 1. Proposed Change.

I propose a relaxation of the review and merge requirements, limited to cases where changes are contained, comprehensively validated, and provably limited to explicitly validated deployment scenarios.

The remainder of the RFC details the following:

1. Process changes we can make to more efficiently merge “safe” optimization PRs
2. How we can systematically ensure that an eligible PR is “safe to merge”

#### 1.1: One-Touch Merge (“LGTM”)

Reviewers often need to revisit PRs multiple times throughout review: one pass to leave comments, another pass to approve and enable CI, additional passes to rerun/retry CI, and possibly a final force-merge request must be made.

For contained model optimizations, much of this can be prevented:

- If no comments are required, a single reviewer action approves the PR and starts an automated procedure to merge: CI runs and executes an automated procedure of rebase and retry until the PR goes in (up to 3 times) or a fallback force-merge is automatically requested.
- If only trivial edits or conflict resolution are requested, a reviewer may submit a “conditional approval” on the requested changes. After the changes are made the automated merge procedure begins. The “conditional approval” persists through rebase, conflict resolution, acceptance of reviewer-suggested edits, and changes localized to tests/, provided the changes preserve the approved behavior. An amount of trust is delegated to the contributor in the event of a conditional approval.

The CI runs should hopefully not be an obstacle in these cases as changes should be contained to localized files/directories which will mitigate the range of detected test cases.

In cases when a maintainer decides nontrivial changes are needed, the maintainer closes or demotes the PR and the contributor is responsible for updating and re-opening or re-requesting review.

#### 1.2: Primary-Maintainer Merge

One common issue that blocks merge is when the primary maintainer of a feature makes a small PR to update an aspect of that feature which they themselves have the most expertise with. In such cases it can be difficult to find a dedicated second reviewer who can confidently approve the changes.

To resolve this I propose we permit core committers to merge their own PRs, provided they follow the validation and isolation requirements of the fast-track specification below.

#### 1.3: Priority Review

I propose that eligible fast-track PRs be automatically labelled as such, and that maintainer review is prioritized towards fast-track PRs. This will involve slack integration, an automated tracking dashboard, and escalation when fast-track-eligible PRs are “stale” for more than N=7 days.

Because fast-track model optimizations are targeted towards a specific model (or small family of models), there should be a small set of assigned reviewers who can be held more directly responsible for triaging fast-track PRs. This burden will be opt-in, but should not be too heavy as triaging fast-track PRs should be very easy by design.

#### 1.4: PR Template

Similar to the [PR-Checklist](https://github.com/vllm-project/vllm/pull/57084) change which updated the PR template, there will be a dedicated template for fast-track PRs which formalizes the requirements and streamlines review. PR will follow the RFC within a few days.

### 2. Validations and Confidence

#### 2.1: Isolation of Code Changes

The primary risk of merging code tailored to optimize one use-case is impact to other deployments. Performance regressions, breaking other features, or correctness violations are all common side-effects of a seemingly harmless change.

The first pillar of raising confidence in safety is _isolation_: limiting code changes to a specific deployment scenario.

I propose the initial prototype require that fast-track changes be limited to “flat model” subdirectories (e.g. all diff is within a folder such as `vllm/models/deepseek_v41/nvidia`).

#### 2.2: Bounded Impact on Deployments

Optionally, changes can be further gated programmatically (e.g. a kernel fusion only enabled at small input sizes). The more isolated a change is, the smaller the potential impact. By placing rigid bounds on the changes, we can limit the impact to a very small set of deployment scenarios which can be tested explicitly.

Naturally, limiting the scope in this way also reduces the net improvement per PR: other excluded scenarios might benefit from the changes as well. This may also increase code duplication and redundancy. However, it creates a clear path toward improving stability and scalability as we accelerate the pace of optimizations with coding agents.

[TensorRT-LLM has merged a design](https://github.com/NVIDIA/TensorRT-LLM/pull/19056) which isolates/replicates model code for each model, hardware, and parallelism triple (e.g. gpt-oss TP1 on SM103). This is a fairly extreme flavor of replication, but it highlights an architectural direction we can move towards for if the initial results from vLLM’s flat-model design prove sufficiently maintainable.

#### 2.3: Explicit Validation of Affected Deployments

Once an isolated and bounded deployment is established for a contribution, what remains is to run sufficient validations against a representative set of configurations.

Since reviewers are still involved in the process, there is some flexibility in the extent of validation: some changes might warrant testing every parallelism configuration or testing on multiple hardwares, while others might be sufficient with only a single parallelism/HW configuration. If a contribution is not sufficiently validated, reviewers may request additional validation and approve conditionally (see 1.1) or they may demote the PR from the fast-track and/or close it as necessary. Submissions following the fast-track accept a higher evidence standard and a higher risk of reviewers closing low-quality PRs wholesale without discussion.

To codify expectations, I propose the following requirements:

- **E2E Correctness Evaluation:**
  - `GSM8k` and `GPQA-Diamond` be run for all proposed model optimizations
  - A long-context benchmark such as `MRCR v2 256K (8-needle)` and/or a long-generation benchmark such as `AIME25`/`AIME26` (pass\@1 avg1) should be included when changes affect sensitive token-mixing operations such as FMHA or SSM.
  - A “hard” correctness benchmark such as `AIME25`/`AIME26` (pass\@1 avg4+) or `HLE` or `TerminalBench (any)` should be included when changes include a “high correctness risk factor”: low-level CUDA C++ or PTX/SASS or equivalent, or new communications kernels for which race conditions cannot be ruled out by formal verification.
  - All correctness evaluations should use at least “high” reasoning effort, where supported. All evaluations should compare against a known “good” baseline, either referencing official vendor reported numbers or those from a vLLM reference run. Running your own baseline on the vLLM main branch to compare against is acceptable. Measured scores should exceed or match the reference within one percentage point (absolute delta, after accounting for run-to-run variance).
  - Correctness validation runs should ensure that the deployment scenario is reproduced. For example, if adding a small-batch fusion, GSM8k should run at a correspondingly low batch size (to trigger the change and contribute coverage to the diff).
- **Kernel Correctness Evaluation:**
  - New kernels should be exhaustively validated for correctness on a broad range of inputs. This includes multiple possible input shapes as well as input distributions.
    - If bit-exactness is not maintained, it should be justified and validated with a strict tolerance.
    - If quantization or lossy optimizations are used, this is considered a “high correctness risk factor” and warrants additional E2E validation (see above).
  - For low-level CUDA kernels, include [compute-sanitizer](https://developer.nvidia.com/compute-sanitizer) tests such as `memcheck`, `racecheck`, and `synccheck` sanitization are required where applicable.
  - Note, committed tests should be a small subset of validation performed during development. These should exist strictly for regression checks, not for comprehensive reproduction. We do not want to run exhaustive correctness testing in CI every time we make a model change.
- **Performance Evaluation:**
  - PRs should include a “headline” speedup on a specified “target” workload. This is required but not sufficient for performance validation.
  - Performance change should also be swept across the range of bounded-affected-workloads: multiple concurrencies should be benchmarked almost always, as well as high/low sequence length when applicable. Any throughput or latency regression exceeding 1% for any measured configuration is unacceptable and disqualifies the PR from the fast-track.
    - For MoE models, ensure real prompts (such as speed-bench throughput_8k) are used so that expert balance is realistic.

### Feedback Period.

Feedback period will continue for 12 days, ending on Oct 13. @benchislett will request maintainer review and open a candidate PR with a reference implementation of the proposed design.

### CC List.

@njhill @LucasWilkinson @yewentao256 @WoosukKwon @mgoin @vllm-project/vllm-committers @xinli-sw

### Discussion Areas

1. Do we want to further solidify the requirements from sec. 2.3 into a structured requirement matrix? Or do we want to leave it flexible for reviewers to decide on a per-PR basis if validation is sufficient?
2. How strict should we be about isolating code changes? Are minor one-liner integration patches in core files acceptable, or do we require rigid file-structure-based guarantees about isolation? Can we build static linting tooling (possibly AI powered) that can enforce or verify that changes are limited to the claimed workloads?
3. Is committer self-merge a good idea, or is it too permissive?
4. How much control are we comfortable delegating to automated verification systems, versus committer judgement? Should we be more/less aggressive in automating merge workflows?

---

## comments

**ZJY0516** (2026-10-02T01:25:35Z):

> One common issue that blocks merge is when the primary maintainer of a feature makes a small PR to update an aspect of that feature which they themselves have the most expertise with. In such cases it can be difficult to find a dedicated second reviewer who can confidently approve the changes.

> To resolve this I propose we permit core committers to merge their own PRs, provided they follow the validation and isolation requirements of the fast-track specification below.

> Is committer self-merge a good idea, or is it too permissive?

I personally think we should need another reviewer to make sure the coding style is not too bad.

**benchislett** (2026-10-02T01:42:33Z):

I think that we should have enough trust in our core team to ensure code quality does not suffer. If agents are writing the code anyways, it often only gets one look anyways (from the assigned reviewer). The idea here is that core committers should still self-review, but wouldn’t need to get a second opinion before merging because the usual concerns about side-effects and risk to other deployments wouldn’t apply to fast-track PRs

**benchislett** (2026-10-02T01:43:08Z):

I definitely understand if, even given this caveat, we think that this is too much power to bestow on all existing committers.

**benchislett** (2026-10-02T01:48:35Z):

Adding some helpful feedback from a colleague: agents love to hide their features behind env vars, which adds complexity to deployments and risks explosion of the configuration space.

The pattern I wish to propose here is _isolate and dispatch_: for instance if a contribution is only beneficial for `batch <= N tokens`, it should either:

- dispatch the optimized path at runtime when the batch size is sufficiently small, falling back to prior behaviour otherwise (preferred)
- only take effect when the runtime is guaranteed to always use small enough batches, such as via `max_num_batched_tokens <= N` (not ideal, but permissible for optimizations which may need an alternate weight layout for instance).

**yewentao256** (2026-10-02T14:14:19Z):

For some certain domain that doesn’t have a second reviewer. Eg. Batch invariance, only me review the coming PRs from community.
In this case, I think if I can self-merge my update to batch invariance would be good.

**benchislett** (2026-10-02T16:54:18Z):

My initial proposal for this RFC is to limit the fast-track to PRs contained specifically to model optimization code, but this is not a fundamental requirement. In theory any contribution that constrains its changes to a strictly-isolated code region would be eligible, if we want to make it so. I think we should try a pilot focusing on model optimizations first, and can expand the constraints if it proves valuable.

In the specific case of reviewer self-merge, I’m generally in favor of making this the default policy (committers can merge their own code), but last time I brought this up it was deemed a bit too risky. So I think having the guardrails of the fast-track makes it a good way to test out the idea before committing fully

**ZJY0516** (2026-10-05T08:30:34Z):

> committers can merge their own code), but last time I brought this up it was deemed a bit too risky

yes, especially with coding agent. My agent tried to directly merge my PR to main several times before

**vadiklyutiy** (2026-10-09T00:40:36Z):

A few thoughts on the proposed review and merge process:

One-touch merge: I’m generally in favor. We already often merge with a single approval, and automating the remaining steps would be convenient. My main concern is whether automatic CI runs and retries could cause unnecessary runs and increase CI load.

Conditional approval: Do new commits currently dismiss existing approvals in our repo? My understanding is that they don’t, so I’d like to understand what conditional approval would change compared to the current workflow.

Self-merge: I’m somewhat hesitant about this. Having the same person make and review a change often works, but just as often it doesn’t. Authors may be under pressure from their own deadlines or other commitments, and it’s natural to be more forgiving of your own code. An independent reviewer brings a fresh perspective and can help catch things the author might overlook or be more willing to accept. I’d prefer to keep a second person involved, even for core committers.

**vadiklyutiy** (2026-10-09T02:40:05Z):

A separate thought on reducing reviewer workload, beyond the specific proposals in this RFC.

I agree with the premise: agents generate code faster than we can review it. But I don’t think we can solve this by continuing to put human reviewers up against coding agents. It feels like we’re cavalry trying to fight tanks by improving our sabers. We need to build our own tanks—and in this case, that means using agents for review too.

For quite a while now, the first thing I do before reviewing a PR is ask an agent to review it. In most cases, it comes back with several useful findings. We’re not ready to fully entrust code review to agents, but they can already save human reviewers a substantial amount of time.

I see two possible directions:

1. An automated cloud-based review agent integrated with GitHub. We’ve tried several of these, but for some reason they haven’t stuck and were eventually disabled. I’ve asked about this a few times in Slack over the past few months but haven’t received an answer. I’d like to understand what didn’t work and whether we can address it.

2. Encourage contributors to run a separate agent session to review their changes before requesting human review. We already have a good skill covering the main aspects of code review, so there’s something concrete to build on. The separate session matters: it should review the changes with fresh context. What I have in mind is a local agent with a local checkout of vLLM and access to a GPU, so it can actually run GPU checks—not just someone pasting a diff into the free version of ChatGPT.

I think putting more effort into agent-assisted review is essential if we want reviewer capacity to keep up with agent-generated contributions.

**Isotr0py** (2026-10-09T03:22:25Z):

> An automated cloud-based review agent integrated with GitHub. We’ve tried several of these, but for some reason they haven’t stuck and were eventually disabled. I’ve asked about this a few times in Slack over the past few months but haven’t received an answer. I’d like to understand what didn’t work and whether we can address it.

I think the reason we deprecate these cloud-based review agents is the degradation of their review quality, which makes noise during review. We still have a review bot in vllm-omni to review PR there, but I sometimes feel it too noisy and its review comments would become quite long and unreadable (like [vllm-project/vllm-omni#8099 (comment)](https://github.com/vllm-project/vllm-omni/pull/8099#issuecomment-5947349817)).

I’m also using <https://github.com/alibaba/open-code-review> with K3 to assist me reviewing PR, but I feel the agent review quality won’t be much stable and reasonable without multi-turn human feedbacks. 😅

**maeehart** (2026-10-09T15:05:50Z):

Thanks for writing this up. ROCm model PRs have the same problem with reviewer time, so we would like to use the fast track too. The eval guidance has already helped on [#60397](https://github.com/aarnphm/aarnphm.github.io/issues/60397), where the AIME25 run that you asked for came out 1.5 points below the reference, and looking for the cause found a race in the new kernels.

My main question is how a PR qualifies and who decides that. The RFC says that eligible PRs are labelled automatically and that reviewers can demote a PR, but it does not say what the labelling checks or which reviewers can demote. That is why, on Discussion Area 1, I think that a written requirement matrix is better than a decision per PR, because the author then knows before the review which evals the PR needs, and every hardware folder gets the same bar.

On who decides, would the assigned reviewers from section 1.3 be the code owners of the folder that the PR changes? The `/vllm/models/*/amd` folders already have their own entry in `.github/CODEOWNERS`, so for them this would also settle who triages and who can demote.

On Discussion Area 2, most ROCm model PRs also change files outside the model folder. Between 19 Sep and 9 Oct, 97 ROCm PRs changed a `vllm/models/*/amd` folder, and only 14 of them changed nothing else. Three kinds of change come up:

1. Most of them add a test in `tests/`, and some add a CI step in `.buildkite/`. For 20 of the 97 PRs these were the only changes outside the model folder, so allowing them would more than double the PRs that meet the isolation rule.
2. A new AITER kernel needs an AITER release and the `AITER_BRANCH` bump in `docker/Dockerfile.rocm_base`, which already go in their own PR, such as [#59794](https://github.com/aarnphm/aarnphm.github.io/issues/59794). The model PR then often adds a wrapper in `vllm/_aiter_ops.py`.
3. Several open PRs, such as [#60397](https://github.com/aarnphm/aarnphm.github.io/issues/60397) and [#60683](https://github.com/aarnphm/aarnphm.github.io/issues/60683), add mono decode kernels for different models, and each one carries its own adapted copy of the same FlyDSL helpers. One shared copy would sit outside every model folder.

Can the tests, the CI step and the wrapper be part of the fast-track PR when they serve only that model folder? Also, should shared code like the mono decode helpers go in a normal PR first?

