---
title: '57084'
pageLayout: default
slug: thoughts/inference/vllm-prs/57084
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57084.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57084](https://github.com/aarnphm/aarnphm.github.io/issues/57084) \[Docs\] Add PR checklist skill for coding agents

[vllm-project/vllm#57084](https://github.com/vllm-project/vllm/pull/57084) · benchislett · MERGED · created 2026-09-15T22:22:05Z · merged 2026-09-30T21:47:29Z

## Overview

(DISCLAIMER: I USED THE NEW SKILL TO HAVE GPT-6 AUTHOR THE PR DESCRIPTION BELOW, COMMIT THE CHANGES AND OPEN THIS PR. TREAT IT AS A REVIEWABLE EXAMPLE ARTIFACT. ALL CODE CHANGES WERE HAND WRITTEN WITH CARE BY ME.)

Add a `pr-checklist` skill for coding agents preparing, reviewing, or revising
vLLM pull requests. Integrate it with the agent contribution instructions and
the PR template.

## Claims

- Provide a structured review workflow covering design fit, compatibility,
  complexity, tradeoffs, validation, test quality, code clarity, and PR evidence.
- Distinguish validation performed for a contribution from tests maintained in
  the repository, emphasizing meaningful coverage rather than test volume.
- Include guidance for human-in-the-loop review and responding to prior feedback.
- Require agents to use the skill before submission through `AGENTS.md`.
- Restructure the PR template around Overview, Claims, Validation, and Details,
  with checklist and AI-assistance disclosure fields.

The intended benefit is better-prepared contributions and less avoidable
maintainer follow-up. This PR does not claim a measured reduction in review
rounds or review time.

## Validation

This change is documentation and agent instructions only. No serving-engine,
model, kernel, dependency, or runtime behavior is modified; runtime tests,
model evaluations, and performance benchmarks are not applicable.

### Diff and structure checks



```bash
git status --short
git diff -- AGENTS.md .github/PULL_REQUEST_TEMPLATE.md
git ls-files --others --exclude-standard .agents/skills/pr-checklist
git diff --check
git diff --no-index --check /dev/null .agents/skills/pr-checklist/SKILL.md
wc -l AGENTS.md .agents/skills/pr-checklist/SKILL.md
```

- Reviewed the two tracked-file diffs and the complete new skill.
- Confirmed the proposed change consists of those three files; research data,
  scraping scripts, plots, and intermediate drafts are excluded.
- `AGENTS.md` has 159 lines and the skill has 153 lines, within the applicable
  instruction-document budgets.
- The tracked-file whitespace check produced no diagnostics.
- The new skill’s whitespace check reports one trailing space on the helper
  inlining instruction. This remains in the proposed diff.
- No Markdown linter or documentation build was run.

### Current-skill behavioral smoke test

A fresh agent context read the current skill and applied it to four
hypothetical scenarios without executing tests or changing files:

1. **This documentation-only PR:** recognized runtime validation as not
   applicable and assessed workflow consistency and documentation.
2. **Reviewing another author’s optimization PR:** requested missing
   configuration, correctness, and serving-performance evidence rather than
   skipping the PR-contents section.
3. **Re-review with an unresolved backend concern:** traced related paths,
   requested appropriate evidence, and identified a stale description.
4. **Intentional all-drafts-rejected regression test:** identified a conflict
   between the expected zero acceptance and the skill’s nonzero-acceptance
   instruction. This remains an outstanding caveat below.

These are qualitative instruction-following checks, not automated regression
tests or evidence of real-world review-time savings.

## Details

### Motivation and research

The drafting process examined the latest 50 vLLM PRs reviewed by each of
`benchislett` and `mgoin`, ordered by the reviewer’s latest submitted review.
The samples contain 97 distinct PRs and 342 nonempty authored review/comment
messages. Recurring themes included compatibility across execution paths,
appropriate ownership and reuse, concise implementations, meaningful rather
than excessive tests, and claims supported by evidence.

A separate exploratory analysis measured additions and deletions under
`vllm/` and compared them with description-based complexity grades. It
reinforced that LOC is a coarse review aid, not a reliable standalone semantic
complexity classification. The skill’s 50/250 bands are judgment-based
guidance, not validated predictive cutoffs.

The research artifacts are local preparation material and are not included
in this PR. The proposed skill was handwritten and iteratively refined by
the human author with AI-assisted research and review.

### Duplicate-work checks

Open-PR searches were run on September 15, 2026:



```bash
gh pr list --repo vllm-project/vllm --state open \
  --search '"pr-checklist"' --limit 30 --json number,title,url,headRefName
gh pr list --repo vllm-project/vllm --state open \
  --search '"skill" "review"' --limit 30 --json number,title,url,headRefName
gh pr list --repo vllm-project/vllm --state open \
  --search 'checklist in:title' --limit 30 --json number,title,url
gh pr list --repo vllm-project/vllm --state open \
  --search 'skill in:title' --limit 30 --json number,title,url
gh pr list --repo vllm-project/vllm --state open \
  --head 'benchislett:skill-pr-checklist' --json number,title,url
```

The broad searches returned unrelated PRs mentioning review or checklist
usage. Targeted title searches found checklists for tool-calling models and
server hardening, and skills for CI and installation; none appeared to
duplicate this general contribution-review workflow. No existing open PR
was found for this branch. No associated issue was supplied, so issue-specific
duplicate checks were not applicable.

### AI assistance

AI assistance was used for GitHub review-history collection, qualitative
categorization, exploratory LOC analysis, draft review, behavioral smoke
checks, and preparation of this description. The human submitter remains
responsible for reviewing and understanding every changed line.

---

<details>
<summary>Pull Request Checklist</summary>

- [x] I used vLLM’s `/pr-checklist` skill.
- [x] AI assistance was used during the creation of this PR.
- [x] **Design Fit:** Adds a focused skill and integrates it into the contribution workflow without runtime changes.
- [ ] **Testing and Validation:** Documentation checks and behavioral smoke cases were run; the rejection-test instruction caveat and whitespace diagnostic remain outstanding.
- [ ] **Code Quality and Style:** Review the helper instruction and invocation/disclosure wording noted above; no lint pass has been claimed.
- [x] **Pull Request Contents:** Describes the purpose, scope, evidence, duplicate-work checks, limitations, and AI assistance.

</details>

---

## comments

**mergify** (2026-09-15T22:22:42Z):

Documentation preview: [https://vllm—57084.org.readthedocs.build/en/57084/](https://vllm--57084.org.readthedocs.build/en/57084/)

**faradawn** (2026-09-16T18:50:30Z):

Love the statement that it is “mandatory for agents, optional for humans”.

Some reviewer pointed out that agent generated PR descriptions tend to be long. Do we want to “limit to 100-200 words”?

LGTM overall

**benchislett** (2026-09-16T18:55:15Z):

@faradawn I indeed share the frustration of overly wordy PR descriptions. However, we don’t want to suppress agents from sharing relevant drawbacks.

My compromise here was to enforce header sections that are length-constrained:

> Ensure the PR description has a short 1-2 line summary of the changes to facilitate quick understanding by reviewers

Perhaps I should extend this to explicitly mention the “claims” section.

For Validation and Notes sections, I thought it would be better to let these sections be wordy — we often do want to see extensive validation results, and a detailed root-cause trace, for tricky issues.

I thought about requiring that these be sorted into _details_:

<details><summary>example</summary>
<p>

some very long validation content could go in here

</p>
</details>

But I’m not sure if that adds much value if there’s not anything else in the description we’re trying to scroll _towards_.

**faradawn** (2026-09-16T20:19:57Z):

Good point that we want the agents to share potential drawbacks. I like the idea that of a “short 1-2 line summary”.

**faradawn** (2026-09-21T23:09:38Z):

Tagging survey respondents who flagged this. Please react 👍 or comment.

@AndreasKaratzas — you flagged that agent PR have long text. Does the “1–2 line limit” fix that?

@tlrmchlsmth — “way too much text everywhere.” Does this PR address your concern?

Also @njhill, @bbrowning, @DarkLight1337, @tjtanaa, @mgoin, @esmeetu, @aoshen02, @penli9 - 👍 or comments welcome.

Aiming to merge by Friday Sep 25.

**AndreasKaratzas** (2026-09-22T00:35:21Z):

My prompt to the agent usually is “write 3-5 bullet point list with your modifications followed up by a 3-6 sentence mini-paragraph with the story and motivation behind these changes” (including buildkite link or regression PR if the PR is targeting a CI regression fix). For kernel PRs, I do like the occasional print screen of the traces as well, to ensure that this was not 100% vibe coded, contrary it was validated and the author rendered the traces before and after their modifications. Also table about gains and perf evals are good, but again a prompt could be limit this to one lm eval table and a bench table that contains average and stats across 10 bench runs.

**njhill** (2026-09-28T17:31:55Z):

@benchislett just in case it wasn’t obvious, most of my comments are suggesting additions rather than replacements.

Also I just had another thought that we could add a specific directive to ensure cpu<span>&larr;</span>>gpu checks aren’t introduced on the main model execution thread (though we now have the checker in CI, some still slip through since not all configurations are exercised in CI).

**benchislett** (2026-09-28T23:05:18Z):

Got very busy this weekend. Will update tomorrow. Thanks very much everyone for feedback!

**benchislett** (2026-09-30T14:49:48Z):

Going to trigger CI just to see what it wants to run. Ideally nothing, since there are no code changes

**benchislett** (2026-09-30T14:49:56Z):

/ci run

**github-actions** (2026-09-30T14:50:17Z):

✅ Triggered [Buildkite CI #92177](https://buildkite.com/vllm/ci/builds/92177) for commit `a50019c97914`.

**vllm-agent** (2026-09-30T18:31:38Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 0 test steps (0 jobs) instead of 0 (0 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 0 (0)         | 0 (0)    | 0 (0)      | 0 (0)     |
| AMD mirrors            | 0 (0)         | 0 (0)    | 0 (0)      | 0 (0)     |

Docs-only change: today’s rules run no tests.

<details><summary>Selector would run (0)</summary>

none

</details>

<details><summary>Would skip (today's rules run them) (0)</summary>

none

</details>

<details><summary>Would add (today's rules do not run them) (0)</summary>

none

</details>

<details><summary>AMD mirrors: would skip (0)</summary>

none

</details>

<details><summary>AMD mirrors: would add (0)</summary>

none

</details>

#### CI results (2026-09-30 18:52 UTC)

0 passed, 0 failed, 0 pending.

No failures to judge.

<sub>4 changed files · base `2df122e65e` · head `a50019c979` · Python record: not used (/tmp/ci-infra-selector/buildkite/ci_selector/coverage-data/table.json.gz is table version 5, expected 7; re-merge it from the raw recordings) · kernel record: table [`866fa13`](https://github.com/aarnphm/aarnphm.github.io/commit/866fa130fa) (build 92059), map [`866fa13`](https://github.com/aarnphm/aarnphm.github.io/commit/866fa130fa) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 0 optional steps the selector would also run</sub>

