---
date: '2026-07-21'
description: the PyTorch, model, vLLM, and interview core for Inferact
id: core
modified: 2026-07-23 03:45:29 GMT-04:00
tags:
  - cs
title: Inferact inference core
created: '2026-07-21'
published: '2026-07-21'
pageLayout: default
slug: hinterland/prep/inferact/core
permalink: https://aarnphm.xyz/hinterland/prep/inferact/core.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# core map

The shortest route is tensor semantics, inference components, model execution, vLLM state ownership, performance reasoning, system design, and one deep project. Each layer depends on the previous one.

```mermaid
flowchart LR
  Tensor["tensor metadata"] --> Ops["stable tensor operations"]
  Ops --> Model["model components"]
  Model --> Runtime["inference runtime"]
  Runtime --> VLLM["vLLM request lifecycle"]
  VLLM --> Serving["distributed serving"]
  Serving --> Design["system design and deep dive"]
```

## priority 0: tensor metadata

Every PyTorch tensor carries shape, stride, dtype, device, storage offset, layout, and autograd state. Most coding failures in this role-shaped surface begin here.

| concept                    | interview bar                                                                                 | typical failure                                                        |
| -------------------------- | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| shape                      | name batch, sequence, head, beam, vocabulary, block, and feature axes before coding           | a broadcast succeeds with the wrong semantic alignment                 |
| stride and storage offset  | determine whether an operation is a view, a copy, or incompatible with `view`                 | transpose followed by `view` fails or `reshape` hides an allocation    |
| aliasing                   | know when input and output share storage and whether mutation is legal                        | an in-place cache update changes another logical view                  |
| indexing                   | distinguish basic-index views from advanced-index copies; use gather and scatter deliberately | hidden allocations or batch mixing                                     |
| dtype                      | select accumulation precision and know when casting is required                               | fp16 logits overflow or quantization scales apply along the wrong axis |
| device and synchronization | keep tensors colocated and identify host-device synchronization                               | `.item()`, printing, or a device copy stalls the decode loop           |

Read [tensor views](https://docs.pytorch.org/docs/stable/tensor_view.html), [`Tensor.view`](https://docs.pytorch.org/docs/stable/generated/torch.Tensor.view.html), [broadcasting semantics](https://docs.pytorch.org/docs/stable/notes/broadcasting.html), and [`torch.gather`](https://docs.pytorch.org/docs/stable/generated/torch.gather.html).

## priority 1: stable inference operations

Own these from an empty editor:

- masked fill with a defined all-masked-row contract
- `logsumexp`, `softmax`, and `log_softmax` in a stable accumulation dtype
- top-k and top-p filtering
- deterministic sampling with an explicit `torch.Generator`
- beam parent and token selection with `gather`
- last-valid-token gather for ragged batches
- in-place cache writes with unchanged storage addresses
- vectorized block-table lookup

For beam search, use `log_softmax` rather than `softmax().log()`. For sampling, require every probability row to be finite, nonnegative, and have positive mass. For low-precision logits, accumulate normalization in fp32 unless the contract explicitly says otherwise.

Read [`softmax`](https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.softmax.html), [`log_softmax`](https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.log_softmax.html), [`topk`](https://docs.pytorch.org/docs/stable/generated/torch.topk.html), and [`multinomial`](https://docs.pytorch.org/docs/stable/generated/torch.multinomial.html).

## priority 2: modules and inference state

Know the boundary among parameters, buffers, ordinary attributes, and request state.

| state                 | owner                     | example                                                |
| --------------------- | ------------------------- | ------------------------------------------------------ |
| parameter             | model                     | projection or embedding weight                         |
| persistent buffer     | model state dictionary    | precomputed RoPE frequency table when persistence fits |
| nonpersistent buffer  | module device/dtype state | derived lookup table that should move with the module  |
| request state         | runtime                   | sequence length, block table, sampling parameters      |
| mutable cache storage | cache manager and worker  | K/V pages and slot mapping                             |

`model.eval()` changes module behavior such as dropout. `torch.no_grad()` disables gradient recording. `torch.inference_mode()` also removes view tracking and version-counter overhead, with stricter rules on later autograd use. These mechanisms solve different problems.

Read [`nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html), [module notes](https://docs.pytorch.org/docs/stable/notes/modules.html), and [autograd grad modes](https://docs.pytorch.org/docs/stable/notes/autograd.html).

## priority 3: model implementation ladder

Use [[hinterland/prep/inferact/model-builds|the model-build lane]] as the primary coding curriculum. The goal is to turn a config, diagram, or paper fragment into a complete PyTorch module, then connect that module to inference-runtime state. The P-series drills repair individual mechanisms when a full build exposes a gap. [[hinterland/prep/inferact/pytorch-practice|The PyTorch practice bank]] tests the repaired mechanism under a fresh contract, debugging case, or runtime transform. [[hinterland/prep/inferact/gpt-lab|The tiny GPT lab]] materializes M01 and the simple homogeneous-cache M03 path as a runnable CPU model and test matrix; the advanced M03 path adapts its parity invariants to mixed cache lengths. Its prefix-cache exercise sits downstream as a separate runtime owner over the functional model cache, so it does not enlarge M03.

### dense decoder block

Be able to implement and explain:

1. token embedding
2. RMSNorm
3. Q, K, and V projections
4. RoPE at the correct positions
5. grouped-query attention
6. output projection and residual
7. second RMSNorm
8. SwiGLU MLP and residual
9. final norm and logits projection

The shape ledger should include:

```text
hidden: [batch, tokens, hidden_size]
q:      [batch, query_heads, tokens, head_dim]
k, v:   [batch, kv_heads, tokens, head_dim]
scores: [batch, query_heads, query_tokens, kv_tokens]
logits: [batch, tokens, vocabulary]
```

Use [`scaled_dot_product_attention`](https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) as the optimized reference boundary. Know that its dropout argument remains active when greater than zero, even when the surrounding module is in evaluation mode.

### inference variants

| variant                 | mechanism to own                                                            | runtime pressure                                                 |
| ----------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| grouped-query attention | more Q heads than K/V heads; queries share K/V groups                       | lower KV memory and repeated-head indexing                       |
| mixture of experts      | top-k routing, capacity, expert dispatch, weighted combine                  | load skew, all-to-all, expert parallelism, quantized expert GEMM |
| multimodal              | media processing, encoder outputs, placeholder-token merge, encoder cache   | TTFT variance, cache identity, large visual-token counts         |
| hybrid attention/SSM    | attention KV plus recurrent state with different cache semantics            | hybrid cache grouping, alignment, prefix validity                |
| diffusion serving       | iterative denoising with repeated model calls and scheduler-dependent state | batch evolution, graph capture, latency and memory reuse         |
| speculative decoding    | draft proposals, target verification, acceptance, rollback, bonus token     | lookahead cache slots, batch dependence, correctness             |

The live Inferact role explicitly values paper-to-code implementation. For an unfamiliar architecture, first write the config invariants, module tree, state and tensor contracts, and hidden tests. Then implement a reference path and identify which vLLM interfaces the architecture stresses.

The implementation progression is:

```text
minimal causal LM
  -> Llama-style GQA, RoPE, RMSNorm, and SwiGLU
  -> functional prefill and decode cache
  -> vLLM-shaped flattened-token model port
  -> MoE and multimodal variants
  -> hybrid recurrent state or diffusion model
  -> strict weight loading and paper-fragment capstone
```

Read the official [PyTorch transformer-building-blocks tutorial](https://docs.pytorch.org/tutorials/intermediate/transformer_building_blocks.html) and [vLLM basic-model implementation guide](https://docs.vllm.ai/en/v0.17.0/contributing/model/basic/) after attempting the reference build.

## priority 4: PyTorch execution internals

The interview bar is a mechanistic map, not memorizing every dispatcher class.

```mermaid
flowchart LR
  Python["Python module and tensor code"] --> ATen["ATen operator"]
  ATen --> Dispatch["dispatcher and backend selection"]
  Dispatch --> Kernel["CPU, CUDA, Triton, or custom kernel"]
  Python --> Dynamo["torch.compile capture and guards"]
  Dynamo --> Inductor["Inductor lowering and fusion"]
  Inductor --> Kernel
  Kernel --> Graph["optional CUDA graph capture and replay"]
```

Know these tradeoffs:

- `torch.compile` specializes under guards and may recompile when shapes, values, or Python control flow violate them.
- graph breaks preserve correctness and lose fusion or whole-graph optimization opportunities.
- custom operators define an opaque boundary when the compiler should call a kernel rather than trace through its implementation.
- CUDA graphs reduce Python and launch overhead by replaying fixed work with stable memory addresses.
- dynamic request batches need padding, piecewise capture, bounded capture sizes, or an eager fallback.
- profiling must separate compilation and warmup from steady-state execution.

Read [`torch.compile`](https://docs.pytorch.org/docs/stable/generated/torch.compile.html), [`torch.export`](https://docs.pytorch.org/docs/stable/user_guide/torch_compiler/export/programming_model.html), [custom operators](https://docs.pytorch.org/tutorials/advanced/custom_ops_landing_page.html), [CUDA semantics](https://docs.pytorch.org/docs/stable/notes/cuda.html), and [PyTorch profiler](https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html).

Use PT31 through PT35 in [[hinterland/prep/inferact/pytorch-practice#compilation, custom operators, and profiling|the execution-internals practice lane]] after the P27 compile probe is clean.

## priority 5: vLLM request lifecycle

Carry one exact owner map:

```mermaid
sequenceDiagram
  participant Client
  participant Frontend as API and tokenizer
  participant Core as EngineCore
  participant Scheduler
  participant Cache as KV cache manager
  participant Worker as GPU model runner
  Client->>Frontend: request
  Frontend->>Core: tokenized engine request
  Core->>Scheduler: enqueue
  Scheduler->>Cache: match and allocate blocks
  Scheduler->>Worker: token budget, block tables, metadata
  Worker->>Worker: prepare tensors and run model
  Worker->>Core: sampled tokens and status
  Core->>Frontend: engine output
  Frontend->>Client: detokenized stream
```

The V1 scheduler’s central invariant is that each request has computed tokens catching up to known prompt, output, and speculative tokens. Scheduling token counts instead of rigid prefill and decode batches permits chunked prefill, prefix hits, lookahead slots, and external KV loads inside one model step.

The owner chain to explain:

1. frontend owns HTTP, request parsing, tokenization, multimodal preprocessing, detokenization, and streaming
2. engine core owns request lifecycle and output coordination
3. scheduler owns waiting and running state plus per-step token admission
4. KV cache manager owns block matching, allocation, reference counts, prefix reuse, and eviction
5. model runner owns persistent batch tensors, input preparation, graph dispatch, forward execution, and sampling boundaries
6. attention backend owns compatible paged-KV metadata and kernel execution
7. distributed layer owns process groups, rank-local shards, collectives, and connector transport

Read the [architecture overview](https://docs.vllm.ai/en/latest/design/arch_overview/), [[thoughts/vllm|the garden vLLM note]], [[thoughts/Continuous batching|continuous batching]], and [[thoughts/paged attention|paged attention]].

The same owner chain supplies the general-programming bridge. Frontend streaming maps to incremental parsers, scheduler state maps to heaps and queues, cache management maps to hashes, tries, allocators, and reference counts, model-runner metadata maps to stable permutations, and connector transport maps to cancellation and distributed lifecycle. [[hinterland/prep/inferact/programming-practice|The programming bank]] tests those structures without hiding them behind tensor APIs.

## priority 6: KV and scheduler mechanics

### KV capacity

For a decoder-only model, begin with:

$$
\text{KV bytes per token} = 2 \cdot L \cdot H_{KV} \cdot D \cdot b
$$

where the factor of two is K and V, the model has `L` layers, `H_KV` KV heads, head dimension `D`, and `b` bytes per element. Then account for tensor or context sharding, block rounding, hybrid cache groups, quantization scales, reserved watermarks, and non-KV runtime memory.

### prefix caching

The model forward produces K/V. The runtime cache manager decides whether immutable physical blocks can satisfy a later request. Publish only complete blocks of `B` tokens. A partial tail remains request-private until it fills a block, because publishing a mutable tail would make cache identity and concurrent ownership unstable.

For a prompt of `T` tokens, cap reusable tokens at:

$$
R = \max\left(0, \left\lfloor \frac{T - 1}{B} \right\rfloor B\right)
$$

For a nonempty prompt, at least one token must run through the model to produce next-token logits. When `T` is exactly block-aligned, this rule recomputes the final block rather than returning a cache with no logits.

Each complete block uses a chained identity:

$$
h_i = H(h_{i-1}, \text{tokens}_i, \text{namespace})
$$

The namespace includes the model and cache-schema revision plus any adapter, multimodal input, prompt embedding, cache salt, tenant boundary, or position origin that can change K/V. The parent hash prevents the same token block after different histories from aliasing. A digest match remains a candidate match; exact canonical identity and tokens must still agree if the implementation promises collision safety.

Acquisition increments a block’s live reference count. Release removes ownership while leaving a zero-reference block resident and reusable. Capacity is measured in physical blocks. When capacity is needed, evict the least-recently-used zero-reference chain leaf. Evicting an ancestor first would leave its descendants resident and unreachable. Two branches with the same prefix should point to the same physical prefix blocks rather than duplicate their K/V.

The correctness oracle is end-to-end: run only the uncached suffix after the longest valid hit, then require its logits and the reconstructed per-layer K/V to match a full uncached prefill. [[hinterland/prep/inferact/gpt-lab|The executable prefix-cache extension]] tests that oracle with exact full-prefix snapshots. The physical-block design on this page is the production follow-up.

### chunked prefill

Decode is usually bandwidth-bound. Prefill is usually compute-heavy. Chunked prefill uses the remaining per-step token budget to mix them, which can increase utilization while protecting ITL. Chunk size and token budget still expose a TTFT, ITL, and throughput frontier.

### disaggregated prefill and decode

Separate fleets can tune TTFT and ITL independently. KV transfer becomes a critical path, a capacity consumer, and a failure surface. Disaggregation does not create throughput for free.

Read [prefix caching](https://docs.vllm.ai/en/stable/design/prefix_caching/), [optimization and chunked prefill](https://docs.vllm.ai/en/stable/configuration/optimization/), [disaggregated prefill](https://docs.vllm.ai/en/latest/features/disagg_prefill/), [[thoughts/KV connector|KV connectors]], [[thoughts/KV offloading|KV offloading]], and [[thoughts/PD disaggregated serving|P/D serving]].

## current vLLM source tour

The local vLLM checkout at `/Users/aarnphm/workspace/vllm` provides a concrete reading route. Read the owner abstractions before the large model-runner file.

| stop | source path                                                                   | question it answers                                                               |
| ---: | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
|    1 | `vllm/model_executor/models/registry.py` and `model_loader/`                  | How does a Hugging Face architecture string become a loaded model module?         |
|    2 | `vllm/distributed/parallel_state.py`                                          | Which ranks and process groups own TP, PP, DP, EP, and context parallelism?       |
|    3 | `vllm/model_executor/layers/linear.py` and `layers/quantization/`             | How do sharded linear layers compose local GEMMs, collectives, and quantization?  |
|    4 | `vllm/model_executor/models/llama.py`                                         | What does one complete dense decoder implementation look like in vLLM?            |
|    5 | `vllm/model_executor/layers/attention/attention.py` and `vllm/v1/attention/`  | Where do the model facade, cache update, metadata, backend, and kernel split?     |
|    6 | `vllm/v1/kv_cache_interface.py` and `vllm/v1/core/kv_cache_manager.py`        | How do logical blocks become physical KV tensors and block tables?                |
|    7 | `vllm/v1/request.py`, `vllm/v1/core/sched/scheduler.py`, and `engine/core.py` | How does continuous batching move requests through states and token budgets?      |
|    8 | `vllm/v1/worker/gpu_model_runner.py`                                          | How does scheduler output become persistent GPU buffers, model input, and output? |
|    9 | `vllm/v1/sample/sampler.py` and `sample/ops/topk_topp_sampler.py`             | How do logits, masks, penalties, random sampling, and TP vocabulary meet?         |
|   10 | `vllm/compilation/decorators.py`, `backends.py`, and `piecewise_backend.py`   | Where does vLLM partition and lower `torch.compile` graphs?                       |
|   11 | `vllm/v1/cudagraph_dispatcher.py` and `vllm/compilation/cuda_graph.py`        | How does a batch choose eager, piecewise, or full CUDA graph replay?              |
|   12 | `vllm/model_executor/custom_op.py`, `vllm/_custom_ops.py`, and `csrc/`        | How does Python reach a registered PyTorch operator and a CUDA kernel?            |

The high-yield chain is:

```text
architecture string
  -> model registry and loader
  -> LlamaForCausalLM
  -> tensor-parallel layers
  -> attention facade and backend
  -> scheduler and logical KV blocks
  -> GPU model runner and physical tensors
  -> logits and sampling
  -> torch.compile partitioning
  -> CUDA graph replay
  -> torch.ops, C++ registration, CUDA kernel
```

For a short preparation window, read stops 4, 5, 7, 8, 10, and 12. Explain the boundary between each pair before opening the next file.

## priority 7: distributed inference

| method | state split                        | dominant communication or cost                       | useful when                                             |
| ------ | ---------------------------------- | ---------------------------------------------------- | ------------------------------------------------------- |
| TP     | weight and activation dimensions   | row-output all-reduce; optional column-output gather | one model needs multiple tightly connected accelerators |
| PP     | layer ranges                       | stage-to-stage activations and pipeline bubbles      | weak cross-node links or uneven stage memory            |
| DP     | replicated engines and separate KV | request routing and optional synchronized features   | capacity grows through independent replicas             |
| EP     | experts                            | token dispatch and all-to-all                        | MoE weights or compute need sharding                    |
| CP     | sequence or KV context dimensions  | attention-specific exchange and reductions           | context length becomes the limiting dimension           |

Use tensor collectives for tensor data. Object collectives serialize through CPU paths and can introduce synchronization, host copies, and asymmetric work. Every rank must enter collectives with compatible order, shape, dtype, and group semantics.

Read [PyTorch distributed](https://docs.pytorch.org/docs/stable/distributed.html), [tensor parallelism](https://docs.pytorch.org/docs/stable/distributed.tensor.parallel.html), [vLLM parallelism and scaling](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/), and [[thoughts/distributed inference|distributed inference]].

## priority 8: performance language

For any optimization, state this tuple:

```text
workload -> baseline -> measured bottleneck -> mechanism -> result -> regression -> correctness bound
```

Use the right serving metrics:

- TTFT for initial responsiveness
- the ITL distribution for per-token jitter and tail stalls
- TPOT for average per-request decode cadence
- end-to-end latency for user completion
- goodput for throughput that meets latency SLOs
- prompt and output tokens per second for phase-specific capacity
- queue time, running and waiting requests, and preemptions for scheduler pressure
- KV utilization and prefix hit tokens for cache behavior
- accepted draft tokens and verifier time for speculative decoding
- transfer bytes, latency, and failure rate for KV connectors

For kernels, state grid, tile, pointer formula, mask, bytes moved, operations, occupancy limiter, and benchmark evidence.

## priority 9: system-design loop

Use this order on every design:

1. define model, workload, arrival process, input and output lengths, SLOs, quality, and hardware
2. estimate weights, KV bytes per token, concurrency, and communication volume
3. draw request and state ownership
4. choose batching, cache, parallelism, and routing policies
5. define overload, cancellation, retry, and failure recovery
6. instrument request SLOs and server causes
7. identify the first two experiments that decide among designs

A diagram without a capacity estimate is decoration. A capacity estimate without a workload distribution is numerology.

## priority 10: technical deep dive

Select one project with the strongest combination of ownership, mechanism, measurements, failures, correctness, and operational evidence. Use two projects only when the second covers a distinct layer.

The primary story should fit this causal chain:

```text
claim -> boundary -> workload -> baseline evidence -> diagnosis -> intervention -> ablation -> correctness -> deployment -> residual risk
```

Prepare an eight-to-twelve-slide deck, a two-page memo, an architecture diagram, a request-lifecycle diagram, one profiler trace, benchmark and ablation tables, a failure ledger, a correctness matrix, a deployment timeline, and an appendix with formulas.

Every claim should survive these prompts:

- how do you know
- what did you personally own
- what was the first wrong hypothesis
- what regressed
- what invariant protected correctness
- what would change under another workload, model, accelerator, or SLO
- which measurement would prove your diagnosis wrong

## reuse map

- PagedAttention: [[thoughts/paged attention|paged attention]]
- continuous batching: [[thoughts/Continuous batching|continuous batching]]
- speculative decoding: [[thoughts/Speculative decoding|speculative decoding]]
- radix and prefix reuse: [[thoughts/radix attention|radix attention]]
- P/D disaggregation: [[thoughts/PD disaggregated serving|P/D disaggregated serving]]
- distributed inference: [[thoughts/distributed inference|distributed inference]]
- KV compression and offload: [[thoughts/KV compression|KV compression]] and [[thoughts/KV offloading|offloading]]
- GPU fundamentals: [[thoughts/GPU programming|GPU programming]]
- FlashAttention: [[thoughts/flash attention|FlashAttention]]
- quantization: [[thoughts/quantization|quantization]]
- structured outputs: [[thoughts/structured outputs|structured outputs]]
- general coding: [[hinterland/prep/inferact/programming-practice|Inferact programming bank]] backed by [[hinterland/prep/nv/core|the NVIDIA core]]
- queues and streams: [[hinterland/prep/bt/08-queueing/notes|queueing]] and [[hinterland/prep/bt/07-stream-algorithms/notes|stream algorithms]]