How much model, how much data, for your compute?

The Chinchilla Law: C ~ 6 N D

The core scaling law says that loss is a power-law function of model size N and dataset size D. If you want the best possible loss for a fixed compute budget C, you should pick N and D so that both contribute equally to the loss floor.

Compute (FLOPs) ~ 6 * N * D
    where N = parameters, D = training tokens

Chinchilla recipe:  D ~ 20 * N     (20 tokens per parameter is the sweet spot)

This is why every modern lab now quotes its models in "tokens per parameter". GPT-3 had ~1.7 tokens/param (way undertrained). Chinchilla had 20 (compute-optimal). LLaMA 2 7B was trained on ~285 tokens/param (heavily overtrained on purpose -- see below).

Interactive: Chinchilla Compute Budget Calculator

Post-Chinchilla: The Overtraining Era

Chinchilla-optimal minimizes training cost. But in production, inference cost dominates -- you pay the model's running cost for every user query, potentially billions of times. Serving a 70B model at 20 tokens/param is vastly more expensive than serving a 7B model at 285 tokens/param, even if the 70B has slightly lower training loss.

Meta's LLaMA family embraced this trade-off. LLaMA 3 8B saw ~1.875 trillion tokens per billion parameters -- almost 100x the Chinchilla recipe. Training cost went up, but inference cost dropped through the floor.

• GPT-3 (2020): ~1.7 tokens/param -- undertrained by Chinchilla standards
• Chinchilla (2022): 20 tokens/param -- compute-optimal
• LLaMA 2 7B (2023): ~285 tokens/param -- deliberately overtrained
• LLaMA 3 8B (2024): ~1875 tokens/param -- extremely overtrained

The Optimizer Lineage: SGD -> AdamW

Training a 100B-parameter model doesn't work with vanilla gradient descent. Modern LLMs all use variants of AdamW. Here's the lineage:

• SGD: one learning rate for everything, no memory. Noisy.
• SGD + momentum: keeps a running average of gradients. Smooths noise and helps roll through flat regions. Still one LR for all params.
• AdaGrad: adapts LR per parameter based on historical gradients. Problem: LR decays to zero eventually because the denominator only grows.
• RMSProp: exponential moving average of squared gradients instead of running sum. Solves AdaGrad's decay problem.
• Adam: combines momentum (first moment) with RMSProp (second moment) plus bias correction. Tends to generalize slightly worse than SGD due to weight decay interaction.
• AdamW: decouples weight decay from the adaptive LR. Applies weight decay directly to parameters instead of folding it into the gradient. This small fix made Adam competitive with SGD on generalization. Standard for every major LLM.

# Standard AdamW hyperparameters for LLM training
beta1, beta2  = 0.9, 0.999    # exponential decay rates
epsilon       = 1e-8          # numerical stability
weight_decay  = 0.1           # applied directly to weights
learning_rate = 3e-4 with cosine decay after warmup
gradient_clip = 1.0           # global norm

Learning Rate Schedules Revisited

Module 7 introduced warmup + cosine decay. Here we go deeper. The formulas:

# Linear warmup, steps 0..warmup_steps
lr(t) = peak_lr * (t / warmup_steps)

# Cosine decay, steps warmup_steps..total_steps
progress = (t - warmup_steps) / (total_steps - warmup_steps)
lr(t) = min_lr + 0.5 * (peak_lr - min_lr) * (1 + cos(pi * progress))

LR rules of thumb for LLMs:

• 100B+ params: 1.5e-4 to 3e-4
• 10-100B: 3e-4 to 4e-4
• 1-10B: 4e-4 to 6e-4
• <1B: 6e-4 to 1e-3

Bigger models want smaller LRs because their loss landscape is narrower and large steps destabilize training.

Gradient Clipping: The Safety Net

Occasionally a batch contains a weird example that produces an enormous gradient. Without intervention, one bad batch can blow up the entire run. Gradient clipping rescales the gradient whenever its norm exceeds a threshold (typically 1.0):

g_norm = sqrt(sum of squares of all gradient elements)
if g_norm > max_norm:
    g = g * (max_norm / g_norm)   # preserves direction, caps magnitude

Precision: FP32, FP16, BF16

Modern training uses mixed precision: most math in low precision (16 bit) for speed, but critical values (master weights, loss accumulation) in FP32.

• FP32: 32 bits, ~7 decimal digits. Safe but memory-hungry and slow.
• FP16: 16 bits, narrow range (~6e-5 to 65504). Fast but causes underflow during backprop -- solved with "loss scaling".
• BF16: 16 bits with the same exponent range as FP32 (but less mantissa precision). No underflow issues; no loss scaling required. Now the standard for LLM training on A100/H100/TPU.

A 70B parameter model in BF16 takes 140 GB just for weights. Add gradients (140 GB) and AdamW optimizer states (560 GB in FP32 for momentum + variance) and you are already at ~840 GB before you include activations or a training batch. This is why parallelism across many GPUs is mandatory (covered in Module 14).

Batch Size and Training Dynamics

LLM batch sizes are measured in tokens (not sequences). Typical: 2-4 million tokens per update.

• Larger batches = less noisy gradients, better GPU utilization, but past a critical size you get diminishing returns.
• If the batch doesn't fit in memory, use gradient accumulation: process smaller micro-batches sequentially, accumulate gradients, and apply one update. Mathematically equivalent to a large batch.

Check Your Understanding

Flashcards (click to flip)