---
title: '60623'
pageLayout: default
slug: thoughts/inference/vllm-prs/60623
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/60623.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#60623](https://github.com/aarnphm/aarnphm.github.io/issues/60623) \[Chore\] Remove vendored code made redundant by Transformers 5.16.1

[vllm-project/vllm#60623](https://github.com/vllm-project/vllm/pull/60623) · hmellor · MERGED · created 2026-10-08T11:48:15Z · merged 2026-10-09T09:29:04Z

## Overview

Now that the minimum Transformers version is 5.16.1, delete vendored configs, a processor and compat shims that upstream Transformers now provides, porting the model code to the upstream config schemas. Also adds a `transformers-upgrade` agent skill so this unvendoring pass happens on every future floor raise.

## Claims

- Deletes 16 vendored configs (`afmoe`, `AXK1`, `cosmos3`, `diffusion_gemma`, `granite4_vision`, `hy_v3`, `hyperclovax`, `kimi_k25`, `lfm2_moe`, `nemotron`, `nemotron_h`, `olmo_hybrid`, `qwen3_5`, `qwen3_5_moe`, `qwen3_next`, and `AIMv2Config` in `ovis.py`), the vendored `Granite4VisionProcessor`, and dead shims for older Transformers versions:
  - the `ALLOWED_LAYER_TYPES` patch;
  - the Transformers v4 rope branches in `gemma3`, `gemma3n` and `modernbert`;
  - the `audio_token` fallbacks in `granite_speech`.
- Model code reads the upstream fields instead:
  - kimi_k25 vision position embeddings and projector size;
  - Cosmos3-Edge `layers_block_type`;
  - Gemma 4 MoE from `num_experts`.
- Fixes A.X-K1 silently ignoring the checkpoint’s YaRN rope scaling. The vendored `AXK1Config` dropped `rope_scaling`, so vLLM ran with plain RoPE at `max_position_embeddings=131072`.
- Fixes `get_processor` for HF-format InternVL checkpoints (e.g. `OpenGVLab/InternVL3-1B-hf`). The vendored processor was registered under the upstream name `InternVLProcessor`, which hijacked lookup and raised `TypeError: ... missing ... 'image_seq_length'`. The vendored class still serves the original `internvl_chat` checkpoints.
- `hyperclovax` and `kimi_k25` register the upstream classes in `_CONFIG_REGISTRY`, so `--trust-remote-code` keeps using them. Their hub remote config is broken (HyperCLOVAX) or uses an old schema (Kimi).
- Adds the `transformers-upgrade` skill, with scripts that:
  - list vendored code upstream now covers (`find_upstreamed.py`);
  - dump the config and the meta-device model vLLM builds, before and after a change (`dump_configs.py`, `dump_model.py`), and diff the two (`diff_dumps.py`).

## Validation

`main` vs this branch, `transformers` 5.19.0. Each registry checkpoint was built on the meta device with dummy weights (`dump_model.py`): ModelConfig sizes, every parameter shape, and every module’s scalar attributes.

| Checkpoint                                                                        | Model diff vs `main`                                             |
| --------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `google/gemma-4-E2B-it`, `gemma-4-26B-A4B-it`, `gemma-4-31B-it`, `gemma-4-12B-it` | 0                                                                |
| `google/diffusiongemma-26B-A4B-it`                                                | 0                                                                |
| `Qwen/Qwen3.5-0.8B`                                                               | 0                                                                |
| `nvidia/Cosmos3-Nano`                                                             | 0                                                                |
| `skt/A.X-K1`                                                                      | YaRN applied: `layers.*.self_attn.scaling` 0.0722 → 0.1309 (x61) |

The resolved configs (`dump_configs.py`) of these checkpoints only gain upstream defaults or rename fields; no value the model code reads changes, A.X-K1 aside:

- Qwen3.5 (0.8B, 4B, 35B-A3B), Qwen3-Next, LFM2, Olmo-Hybrid, Trinity, Minitron, Nemotron-Mini, Nemotron-H (3 checkpoints), Hy3-preview, HyperCLOVAX, Kimi-K2.5/K2.6, granite-vision-4.1, Cosmos3-Edge, Ovis2-1B.

A.X-K1 is 1 TB in bf16, which doesn’t fit on the test host’s disk, so the YaRN fix is checked on [`thkim93/axk1-2layers`](https://huggingface.co/thkim93/axk1-2layers), a 2-layer cut of A.X-K1 with the same rope config. The check compares teacher-forced prompt logprobs over 2048 tokens of English text, with vLLM measured against the HF Transformers implementation (which applies YaRN), 1x B200, bf16. Absolute logprobs are meaningless for a 2-layer model; only agreement with HF matters.

| Tokens    | mean \|Δ logprob\| vs HF, `main` | mean \|Δ logprob\| vs HF, branch |
| --------- | -------------------------------- | -------------------------------- |
| 0-256     | 0.044                            | 0.028                            |
| 256-1024  | 0.052                            | 0.026                            |
| 1024-2047 | 0.058                            | 0.027                            |

With the fix, vLLM sits at the bf16 noise floor across all positions. `main`’s error is twice as large and grows with position, as expected from missing YaRN. A full-model gsm8k comparison is still pending.



```bash
S=.agents/skills/transformers-upgrade/scripts
PYTHONPATH=<main checkout> python $S/dump_model.py before.jsonl <repos>
PYTHONPATH=$PWD python $S/dump_model.py after.jsonl <repos>
python $S/diff_dumps.py before.jsonl after.jsonl
```

Tests (CPU):



```bash
pytest tests/transformers_utils tests/tokenizers_ tests/config/test_model_arch_config.py tests/models/test_registry.py  # 1252 passed, 269 skipped
pytest tests/models/multimodal/processing/test_common.py -k "Cosmos3 or granite-vision or granite-speech or Qwen3.5 or InternVL3 or Kimi-K2.5 or Ovis2-1B or diffusiongemma or gemma-4"  # 54 passed, 6 skipped
```

Tests (1x B200 per shard):



```bash
pytest tests/models/test_initialization.py -k "<the 47 touched architectures>"  # 50 passed
```

---

<details>
<summary> Pull Request Checklist </summary>

- [x] I used vLLM’s `/pr-checklist` skill. (Mandatory for agents, optional for humans).

- [x] AI assistance was used during the creation of this PR.

- [x] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

- [ ] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

- [x] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.

- [x] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**hmellor** (2026-10-08T11:50:37Z):

/ci run

**github-actions** (2026-10-08T11:50:56Z):

❌ This PR is 3 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:6059227011 -->

**hmellor** (2026-10-08T11:51:03Z):

/ci run —allow-stale

**hmellor** (2026-10-08T12:10:16Z):

/ci run —allow-stale

**github-actions** (2026-10-08T12:10:38Z):

✅ Triggered [Buildkite CI #93591](https://buildkite.com/vllm/ci/builds/93591) for commit `ad914270f83f`.

⚠️ This PR is 4 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

**hmellor** (2026-10-08T14:39:39Z):

End-to-end validation of Kimi-K2.5, Kimi-K3 and Qwen3.8. Everything compares `main` (`e161a0798c`) with this branch (`b7e67d2add`) on 8x B200, `transformers` 5.19.0.

| Model                  | Setup | gsm8k `main` | gsm8k branch | ChartQA `main` | ChartQA branch |
| ---------------------- | ----- | ------------ | ------------ | -------------- | -------------- |
| `moonshotai/Kimi-K2.5` | TP=4  | 0.935        | 0.931        | 0.892          | 0.890          |
| `Qwen/Qwen3.8-27B`     | TP=1  | 0.787        | 0.790        | 0.748          | 0.748          |
| `Qwen/Qwen3.5-35B-A3B` | TP=1  | 0.858        | 0.858        | 0.778          | 0.774          |

These are single runs with greedy decoding, so differences of a few tenths of a point are noise.

- **gsm8k:** `tests/evals/gsm8k/gsm8k_eval.py` against `vllm serve`.
- **ChartQA:** 500 samples from the `HuggingFaceM4/ChartQA` test split, relaxed accuracy, `max_tokens=32`, thinking disabled.
- **Qwen3.5-35B-A3B** stands in for the `qwen3_5_moe` code path of `Qwen/Qwen3.8-2.4T-A95B`.

Kimi-K3 (1.56 TB) and Qwen3.8-2.4T-A95B (4.9 TB) don’t fit on one node, so they were checked without weights:

- **Meta-device build:** `.agents/skills/transformers-upgrade/scripts/dump_model.py` records every parameter shape and every module’s scalar attributes. Kimi-K3, Kimi-K2.5, Qwen3.8-2.4T-A95B and Qwen3.8-27B all show 0 diffs between `main` and this branch. For the Kimi models this includes the shared MoonViT tower and projector that this PR touches.
- **`test_initialization`:** `KimiK3ForConditionalGeneration`, `KimiK3MTPModel` and `KimiK25ForConditionalGeneration` pass (3 passed).

`dump_model.py` needed a fix to build the Kimi models, which I pushed in [`34d7d31`](https://github.com/aarnphm/aarnphm.github.io/commit/34d7d3107f): it now sets the model’s default dtype the way the loader does, and keeps modules on the meta device when model code calls `.to(device)`.

**hmellor** (2026-10-08T15:15:37Z):

@Isotr0py the important models have been validated on B200

@DarkLight1337 both of your suggestions have been added to the skill and (where appropriate) actioned in this PR

**hmellor** (2026-10-08T15:24:55Z):

/ci run

**github-actions** (2026-10-08T15:25:20Z):

✅ Triggered [Buildkite CI #93624](https://buildkite.com/vllm/ci/builds/93624) for commit `d99dbdbb590b`.

**hmellor** (2026-10-08T16:05:30Z):

/ci retry

**github-actions** (2026-10-08T16:05:50Z):

✅ Queued 17 failed job(s) for retry in [Buildkite CI #93624](https://buildkite.com/vllm/ci/builds/93624).

**AndreasKaratzas** (2026-10-10T04:14:05Z):

There is a small regression after this in [`:amd: (MI300) Quantization Shard 1` ](https://buildkite.com/vllm/ci/builds/93915/list?jid=01a123cd-5fcc-4967-81e6-40db6d7db57b\&tab=output) but i m gonna put a fix shortly. This PR was definitely worth the regression though 😅

**hmellor** (2026-10-12T10:08:17Z):

Thanks for the fix @AndreasKaratzas, I’ve made an upstream fix too so that the config is properly defined [huggingface/transformers#49518](https://github.com/huggingface/transformers/pull/49518)

