---
title: '58003'
pageLayout: default
slug: thoughts/inference/vllm-prs/58003
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/58003.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#58003](https://github.com/aarnphm/aarnphm.github.io/issues/58003) \[Agents\] Add API compatibility checking skill

[vllm-project/vllm#58003](https://github.com/vllm-project/vllm/pull/58003) · sfeng33 · MERGED · created 2026-09-21T19:44:52Z · merged 2026-09-30T04:59:49Z

## Purpose

vLLM reimplements the OpenAI and Anthropic APIs, and client libraries break when a field doesn’t match the upstream spec. This keeps happening; each of these was a spec mismatch that shipped and needed a fix:

- [#5031](https://github.com/aarnphm/aarnphm.github.io/issues/5031): `logprobs` response did not match the spec
- [#8272](https://github.com/aarnphm/aarnphm.github.io/issues/8272): streamed tool-call chunks broke the Vercel AI SDK
- [#21052](https://github.com/aarnphm/aarnphm.github.io/issues/21052): `tool_choice="required"` with empty `tools`
- [#25054](https://github.com/aarnphm/aarnphm.github.io/issues/25054): `finish_reason` when a tool is called
- [#40399](https://github.com/aarnphm/aarnphm.github.io/issues/40399): missing `tool_choice` / `tools` validation in Responses

More and more frontend PRs are written and reviewed by coding agents. Asked “is this a bug or intended?” or “what should this default to?”, they answer from memory or from vLLM’s own code. But vLLM’s code is the thing under test, so that reasoning approves drift instead of catching it.

**What this PR adds.** A `check-api-compat` skill. Agents load it on their own for any change, review or question touching `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings` or `/v1/messages`. It tells the agent to:

1. **Read upstream, not vLLM.** Sources are, in order: the OpenAI OpenAPI YAML, the installed `openai` SDK types, then the docs. For Anthropic: the Messages docs, then the `anthropic` SDK types. The agent records which source and version answered.
2. **Read both sides of a field in vLLM.** An endpoint → `protocol.py` / `serving.py` / `api_router.py` map points the agent at the code that uses a field, not just where it is declared. A field that is declared correctly but silently ignored still counts as a bug.
3. **Compare on name, type, default and semantics**, plus streaming terminators, error bodies and omitted-vs-`null`.
4. **Classify before reporting.** Each finding is one of:
   - _matches upstream_
   - _vLLM extension_: allowed, e.g. `vllm_xargs`, `min_p`
   - _incompatibility_: reported with the upstream source and the vLLM `file:line`

**Scope.** Agent config only; no runtime change. Same layout as the existing skills ([#53702](https://github.com/aarnphm/aarnphm.github.io/issues/53702), [#54806](https://github.com/aarnphm/aarnphm.github.io/issues/54806)): `.agents/skills/check-api-compat/` with `agents/openai.yaml` for Codex, and a `.claude/skills/` symlink for Claude Code.

---

## comments

**sfeng33** (2026-09-29T22:32:01Z):

Hi @DarkLight1337 @chaunceyjiang, when you have a moment, could you take a look at this PR? It gives coding agents a check-api-compat skill that makes them check OpenAI/Anthropic API behavior against the upstream spec instead of vLLM’s own code, which should catch spec mismatches before they ship.

**DarkLight1337** (2026-09-30T04:39:29Z):

/ci run —allow-stale

**github-actions** (2026-09-30T04:40:07Z):

✅ Triggered [Buildkite CI #92041](https://buildkite.com/vllm/ci/builds/92041) for commit `60aee3d4e387`.

⚠️ This PR is 158 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

**DarkLight1337** (2026-09-30T04:41:15Z):

/ci run

**github-actions** (2026-09-30T04:41:34Z):

✅ Triggered [Buildkite CI #92042](https://buildkite.com/vllm/ci/builds/92042) for commit `8988440f9170`.

**vllm-agent** (2026-09-30T04:53:52Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 0 test steps (0 jobs) instead of 0 (0 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 0 (0)         | 0 (0)    | 0 (0)      | 0 (0)     |
| AMD mirrors            | 0 (0)         | 0 (0)    | 0 (0)      | 0 (0)     |

<details><summary>Selector would run (0)</summary>

none

</details>

<details><summary>Would skip (today's rules run them) (0)</summary>

none

</details>

<details><summary>Would add (today's rules do not run them) (0)</summary>

none

</details>

<details><summary>AMD mirrors: would skip (0)</summary>

none

</details>

<details><summary>AMD mirrors: would add (0)</summary>

none

</details>

#### CI results (2026-09-30 05:40 UTC)

9 passed, 0 failed, 0 pending.

No failures to judge.

<sub>3 changed files · base `834ad456d3` · head `a3af78384a` · Python record: not used (/tmp/ci-infra-selector/buildkite/ci_selector/coverage-data/table.json.gz is table version 5, expected 7; re-merge it from the raw recordings) · kernel record: table [`d882bdd`](https://github.com/aarnphm/aarnphm.github.io/commit/d882bddbea) (build 91957), map [`d882bdd`](https://github.com/aarnphm/aarnphm.github.io/commit/d882bddbea) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 0 optional steps the selector would also run</sub>

