How does a brain of billions of numbers actually learn?

Step 1: Measuring Error with Cross-Entropy Loss

The model outputs a probability distribution over all ~50,000 possible next tokens. Cross-entropy loss measures how much probability mass landed on the correct token.

Loss = -log(P[correct_token])

-log(0.90) = 0.11   good prediction, low loss
-log(0.50) = 0.69   meh
-log(0.08) = 2.53   bad prediction, high loss
-log(0.001) = 6.91  terrible, very high loss

Why the logarithm? Because it converts products of probabilities (for whole sentences) into sums, and because it punishes confident wrong answers far more than unsure ones. A model that's 99.9% sure of the wrong answer deserves a much stronger correction than one that's 51% sure.

Step 2: Backpropagation -- The Blame Game

Now that we have a single number (loss) describing how wrong we were, we need to figure out which of the billions of parameters deserve the blame. This is backpropagation.

Starting from the loss and working backwards through the network, we compute a gradient for each parameter -- a number telling us "if you nudged this parameter up by a tiny bit, the loss would change by X". The gradient is a directional report card: which way to change, and how strongly.

• Output layer: "Why did I prefer 'painting' over 'sunset'?"
• Layer 47: "Which features made me lean toward painting?"
• Layer 10: "How did my attention patterns contribute to those features?"
• Embedding layer: "Were the initial word vectors even pointing in useful directions?"

Every component gets a gradient. The chain rule from calculus lets us combine gradients at each layer by multiplication: local slope at each layer composes into a global gradient for each parameter.

Step 3: Gradient Descent -- Taking Tiny Steps

Once we know the gradient for each parameter, we nudge it in the opposite direction (down the slope of the loss):

W_new = W_old - learning_rate * gradient

The learning rate (LR) controls how big each step is. Typical values for LLM pretraining: 1e-4 to 6e-4. This sounds tiny -- and it is -- but that's the point. If you made each parameter change dramatically on every update, the model would thrash around and forget everything it already learned. Small, consistent nudges accumulate into sophisticated behavior over trillions of updates.

Batch Learning: Averaging Many Examples

We don't update weights after every single example. We process a batch (32 / 64 / 512 / many more sequences in parallel), average the gradients across all of them, and then apply one update.

Three reasons:

1. GPUs love parallelism. A single example barely uses the hardware; a batch of 512 saturates it.
2. Averaged gradients are less noisy. One weird example might pull the model in a bad direction. Averaged with 511 others, the signal survives and the noise cancels.
3. Balanced updates. A batch containing multiple uses of "rise" (sun rises, bread rises, prices rise) produces a single update that respects all the meanings.

There's a limit: beyond the critical batch size, doubling the batch doesn't keep cutting noise in half -- you're better off using those FLOPs for more steps instead. For LLMs, batches of 2-4 million tokens are standard.

Learning Rate Schedules: Warmup + Cosine Decay

A single fixed learning rate is rarely optimal. Modern LLMs use a two-phase schedule:

• Linear warmup (first ~2-5% of steps): LR starts near zero and ramps up to its peak. Reason: early parameters are random; big steps in random directions would destabilize training.
• Cosine decay (rest of training): LR follows half a cosine curve down to a small minimum. Reason: late in training, the model is near a good solution; small, careful refinements are better than large disruptive jumps.

Teacher Forcing and Parallel Learning

During training, the model never uses its own predictions as the next input. Even if it predicts "hat" instead of "mat", we show it the correct answer and move on. This is called teacher forcing.

This turns a sequence of dependent predictions into a single parallel computation. "The cat sat on the mat" yields 6 training signals (one per position) from a single forward pass, thanks to causal masking letting each position attend only to its predecessors.

Scale, Checkpoints, and the Evolution of a Model

GPT-3 trained on ~500 billion tokens. Reading nonstop at 250 words per minute, that's 2,850 years of text. Training cost: ~3,640 petaflop/s-days. Every few thousand steps, the full model state is saved as a checkpoint so training can resume after crashes.

What the model knows at each stage

• 1% trained (5B tokens): weak grammar, bigram-like surface statistics
• 10% trained (50B tokens): basic grammar stable, interchangeable surfaces ("blue" and "green")
• 100% trained (500B tokens): nuanced understanding, cultural associations, rare-word competence

Validation Loss: Detecting Overfitting and Underfitting

We split data into training (used to compute gradients) and validation (held out, used only to measure progress). Three patterns tell us what's happening:

• Healthy learning: train and validation loss both decrease together.
• Overfitting: train loss keeps falling but validation loss flattens or rises -- the model is memorizing training quirks instead of learning generalizable patterns.
• Underfitting: both stay high -- model too small, LR too low, or training too short.

Overfitting is relatively uncommon in modern LLM pretraining because the dataset is so vast and training is so short (usually <1 epoch over the data). It becomes much more relevant during fine-tuning on smaller datasets.

Check Your Understanding

Flashcards (click to flip)