---
date: '2025-10-02'
description: Bharat Venkitesh on scaling laws, optimizer schedules, and transferable hyperparameters
id: '5'
modified: 2026-06-05 15:08:26 GMT-04:00
seealso:
  - '[[thoughts/Transformers|Transformers]]'
  - '[[thoughts/LLMs|LLMs]]'
  - '[[thoughts/vllm|vLLM]]'
tags:
  - ml
  - tsfm
title: lecture five
created: '2025-10-02'
published: '2025-10-02'
pageLayout: default
slug: thoughts/tsfm/5
permalink: https://aarnphm.xyz/thoughts/tsfm/5.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
## learning curves

Cortes et al. fit test and training error against the number of training examples $l$. both curves approach the same asymptotic error $a$ \[@cortes1993learningcurves\].

$$
\epsilon_{\mathrm{test}}(l)=a+\frac{b}{l^{\alpha}},\qquad
\epsilon_{\mathrm{train}}(l)=a-\frac{c}{l^{\beta}}.
$$

> \[!note\] lecture transcription
>
> the original lecture note recorded the training curve as
>
> $$
> \epsilon_{\mathrm{train}}(l)=-\frac{b}{l^{\beta}}.
> $$
>
> that line omits the shared asymptote. the paper’s equation above governs the interpretation.

the 2012 ImageNet paper describes AlexNet, with five convolutional layers and three fully connected layers \[@krizhevsky2012imagenet\]. VGG and ResNet are later architectures, so the citation should name AlexNet rather than those later families.

## scaling laws

Kaplan et al. fit cross entropy loss for autoregressive Transformer language models as a power law in non-embedding parameters $N$, dataset tokens $D$, and training compute $C$ \[@kaplan2020scalinglawsneurallanguage\]. one useful form is

$$
L(N)=\left(\frac{N_c}{N}\right)^{\alpha_N},\qquad
L(D)=\left(\frac{D_c}{D}\right)^{\alpha_D}.
$$

the exponent is the useful quantity. with $\alpha_N\approx 0.076$, doubling $N$ multiplies the fitted loss term by

$$
2^{-0.076}\approx 0.95,
$$

when data and compute are not limiting. the fit applies inside the tested model family and data setup. a tokenizer, architecture, optimizer, or data change requires a new fit.

Hoffmann et al. fit model-limited and data-limited loss together \[@hoffmann2022trainingcomputeoptimallargelanguage\].

$$
\hat{L}(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}.
$$

under the dense training estimate $C\approx 6ND$, their compute-optimal model size and token count both grow at about $C^{1/2}$. Chinchilla used $70$ billion parameters and $1.4$ trillion tokens, or about $20$ tokens per parameter. Gopher used $280$ billion parameters and $300$ billion tokens, or about $1.1$ tokens per parameter. the comparison changed model size and training data at once.

[1909.12673](https://arxiv.org/abs/1909.12673) \[@rosenfeld2019constructivepredictiongeneralizationerror\]  also fit generalization error jointly across model size and dataset size. that paper concerns scaling across runs. warmup and cosine decay concern the learning rate within one run.

## design choices

scaling laws compare runs inside a model family. GLU variants change the feed-forward block \[@shazeer2020gluvariantsimprovetransformer\]. Switch Transformers change the relation between total parameters and active parameters through sparse expert routing \[@fedus2022switchtransformersscalingtrillion\]. these changes alter the cost and quality of each token update, so the old fit becomes the wrong evidence for choosing between architectures.

the [[thoughts/tsfm/tsfm-optimizer.pdf#page=34|optimizer slides]] separate the run budget from the learning-rate schedule. warmup controls the unstable first updates. decay reduces the step size over the remaining budget.

## $\mu$P and hyperparameter transfer

maximal update parameterization makes a small proxy model useful for tuning a wider target model. Yang et al. change initialization, per-parameter learning rates, attention-logit scaling, and output scaling so update sizes remain comparable as width changes \[@yang2022tensorprogramsvtuning\]. many optimal hyperparameters transfer under that parameterization. optimizer choice and data choice still need their own checks.

let

$$
m=\frac{d_{\mathrm{model}}}{d_{\mathrm{model,base}}}.
$$

the Cerebras-GPT implementation used these width changes. its chosen base width and numerical constants were experiment settings rather than universal $\mu$P defaults.

| component                   | standard parameterization  | Cerebras $\mu$P implementation |
| --------------------------- | -------------------------- | ------------------------------ |
| hidden weight variance      | $\sigma_{\mathrm{base}}^2$ | $\sigma_{\mathrm{base}}^2/m$   |
| hidden weight learning rate | $\eta_{\mathrm{base}}$     | $\eta_{\mathrm{base}}/m$       |
| attention score divisor     | $\sqrt{d_{\mathrm{head}}}$ | $d_{\mathrm{head}}$            |
| output logits               | $W_{\mathrm{unemb}}X$      | $W_{\mathrm{unemb}}X/m$        |

see [2304.03208](https://arxiv.org/html/2304.03208v1) \[@dey2023cerebrasgptopencomputeoptimallanguage\]  for the source table and the full parameter classes.

## data parallel training

each data-parallel rank keeps a model replica and receives a different slice of the batch. the backward pass synchronizes gradients across ranks, usually with all-reduce. this works while one replica and its training state fit on each device. parameter or optimizer-state sharding becomes necessary when that state no longer fits.