The superficial alignment hypothesis

Instruction Masking: Where the Loss Actually Goes

Here's the single most important mechanical detail of SFT. Pretraining's loss is computed on every token. SFT isn't.

During SFT, we set the label of every instruction/prompt token to -100, a sentinel that tells PyTorch's cross-entropy loss "skip this position". Gradients only flow from the assistant's response tokens. This is called instruction masking or loss masking.

The math: for a sequence with loss at positions i in response set R,

L_SFT = - (1 / |R|) · Σ_{i ∈ R}  log P(token_i | tokens_<i)

Without masking, gradient capacity would be wasted on "memorize the user's phrasing" — which is not useful and can actively hurt, since it teaches the model to parrot prompts.

Data Formats: Alpaca and ShareGPT

Two formats dominate the open-source ecosystem:

Alpaca (single-turn)

{
  "instruction": "Give three tips for staying healthy.",
  "input": "",
  "output": "1. Eat a balanced diet rich in fruits and vegetables.\n2. Exercise regularly...\n3. Get 7-9 hours of sleep."
}

Three fields: instruction, input (optional context, e.g., a paragraph to summarize), output. Simple, but can't represent conversations.

ShareGPT (multi-turn)

[
  {"from": "human",  "value": "Explain the difference between DPO and PPO."},
  {"from": "gpt",    "value": "<thinking>The user wants a technical comparison...</thinking> DPO eliminates the reward model by..."},
  {"from": "human",  "value": "Which is more stable in practice?"},
  {"from": "gpt",    "value": "DPO is significantly more stable because..."}
]

Can represent arbitrary conversation length, and (critically) allows <thinking>-style chain-of-thought blocks that teach the model to reason before answering.

Data Quality: The Four Sources

Where do high-quality instructions come from? Four primary sources, each with tradeoffs:

Evol-Instruct: making prompts harder on purpose

Xu et al. (2023) pointed out a weakness of vanilla Self-Instruct: synthetic instructions cluster around easy. Evol-Instruct iteratively rewrites prompts to add constraints, deepen reasoning, or increase specificity:

Original:  "Write a function to add two numbers."
Evolved:   "Write a Python function that adds two numbers but rejects
            non-numeric input and handles integer overflow for very
            large values, with type hints and a doctest."

Rejection sampling fine-tuning (RFT)

Llama 3.1 uses this extensively: for each prompt, generate N responses from the current best model, keep only the ones a reward model or verifier says are good, train on those. For math (Yuan et al. 2023): sample solutions, keep only those whose final answer matches ground truth. The model bootstraps its way to better data.

Data Mixing: The Llama 3 Recipe

No real SFT run uses a single source. Llama 3.1's published data mix:

Training Dynamics: Why SFT Uses Tiny Learning Rates

SFT hyperparameters look very different from pretraining:

Pretraining:        lr ≈ 1e-4 to 3e-4,  batch: millions of tokens,  epochs: 1
SFT:                lr ≈ 5e-6 to 2e-5,  batch: 128–256 seqs,       epochs: 1–3
Llama 3.1 SFT:      lr = 1e-5,           ~8,500 steps,              cosine decay

Why 20× lower learning rate? Two reasons:

1. Catastrophic forgetting. The base model holds trillions of tokens' worth of knowledge in its weights. A large learning rate would overwrite it. With a small one, we gently nudge the distribution toward "respond in instruction style" without erasing underlying capability.
2. Small datasets overfit fast. SFT datasets are tiny (tens of thousands to a few million examples). After 2-3 epochs the loss on training instructions becomes near-zero — but held-out quality starts dropping. This is why 1–3 epochs is standard.

The Modern Iterative Pipeline

Llama 3.1 doesn't just run SFT once. It runs six rounds. Each round looks like this:

This is a positive feedback loop. Each round produces a better model, which generates better synthetic data, which trains an even better model. The compounding is why modern open-source models have caught up to proprietary ones on many benchmarks.

Evaluation suites in use

IFEval — instruction following ("respond in exactly three sentences ending with a question"). Deterministic checker.
MMLU / ARC — factual knowledge and reasoning.
GSM8K / MATH — math word problems.
HumanEval / MBPP — code generation with unit tests.
MT-Bench, AlpacaEval — LLM-as-judge open-ended chat.
Chatbot Arena — human head-to-head (gold standard).

Check Your Understanding

Flashcards