SFT taught it how. Preference optimization teaches it what.

Preference Data: The Triplet

Preference data is different from SFT data. Each example is a triplet:

(prompt, chosen_response, rejected_response)

Three sources:

1. Human annotation. Hire humans to compare two outputs and pick the better. Accurate but expensive and slow.
2. AI feedback (RLAIF). Use a strong model (GPT-4, Claude) as the judge. Cheaper, scalable, risks judge bias.
3. Verifiable rewards. For math/code, skip human judgment entirely: did the answer match? Did the unit tests pass? DeepSeek-R1 used this extensively.

The Original Recipe: RLHF with PPO

The 2022-2023 standard. Three stages:

1. SFT: start with an instruction-tuned model.
2. Train a reward model to predict human preferences using the Bradley-Terry loss.
3. PPO: policy generates responses, reward model scores them, PPO updates the policy to maximize reward -- with a KL penalty toward the SFT model to prevent drift.

PPO is powerful but notoriously painful: four models in memory (policy, reference, reward, value), high variance, tricky hyperparameters, easy to diverge. This is why DPO became popular.

DPO: Skip the Reward Model

Rafailov et al. (2023) proved a clever mathematical equivalence: you don't need a separate reward model at all. The preference signal can be extracted directly from the policy itself, by comparing its current probability ratios against a frozen reference model.

For each (prompt, chosen, rejected) triplet:

1. Compute log-prob of chosen under the trained model and the reference model. Call the difference r_chosen (implicit reward for chosen).
2. Compute the same for rejected. Call it r_rejected.
3. Apply the Bradley-Terry model: loss = -log sigmoid(beta * (r_chosen - r_rejected)).
4. Minimize this loss. Gradient pushes the trained model to increase log-prob of chosen and decrease log-prob of rejected.

# DPO loss (simplified)
pi_chosen   = policy.logp(chosen)
ref_chosen  = reference.logp(chosen)
pi_rejected = policy.logp(rejected)
ref_rejected= reference.logp(rejected)

r_chosen   = pi_chosen   - ref_chosen      # implicit reward
r_rejected = pi_rejected - ref_rejected

loss = -log(sigmoid(beta * (r_chosen - r_rejected)))

The beta parameter controls how far the model can drift from the reference. High beta (0.5) = conservative, stays close. Low beta (0.01) = aggressive, allows more change. Default ~0.1.

Interactive: DPO vs PPO Comparison

Preference Comparison Demo

Beyond DPO: SimPO, KTO, ORPO, IPO

DPO inspired a family of variants, each addressing one of its limitations:

• SimPO (Meng et al. 2024): drops the reference model entirely. Uses average log-probability of the response plus length-normalized target margin. Saves memory, matches or exceeds DPO.
• KTO (Ethayarajh et al. 2024): Kahneman-Tversky Optimization. Works with unary feedback (single response rated good/bad) instead of pairs. Useful when you have thumbs-up/down from real users but not paired comparisons.
• ORPO (Hong et al. 2024): Odds Ratio Preference Optimization. Merges SFT and preference training into one stage, no reference model. Saves ~50% of GPU time.
• IPO (Azar et al. 2023): Identity Preference Optimization. Addresses DPO's overfitting when the Bradley-Terry assumption is violated. Useful when preference data contains contradictions.

GRPO and the Rise of Verifiable Rewards

DeepSeek-R1 (January 2025) marked a shift. GRPO (Group Relative Policy Optimization, from the DeepSeekMath paper) eliminates the value network that PPO needs. For each prompt:

1. Generate multiple responses (4-16) from the current policy
2. Score each with a reward function
3. Compute the group's mean reward as a baseline
4. Advantage = response_reward - group_mean
5. Update the policy with a clipped surrogate objective and KL regularization

The real insight was verifiable rewards: for math, check if the final answer matches. For code, run the unit tests. For format, check whether required tags were used. These deterministic rules can't be hacked the way neural reward models can.

R1-Zero showed that starting from just a base model (no SFT!), GRPO with verifiable rewards produced emergent chain-of-thought reasoning. The model learned to "think step by step" as an instrumental strategy for getting more correct answers.

# Verifiable reward for math
def accuracy_reward(response, ground_truth):
    answer = extract_boxed(response)
    return 1.0 if answer == ground_truth else 0.0

# Format reward: encourages <think>...</think> structure
def format_reward(response):
    return 0.1 if has_think_tags(response) else 0.0

The Alignment Tax

Preference optimization is not free. Costs:

• Over-refusal. Model becomes overly cautious ("I can't discuss killing Linux processes..."). XSTest measures this.
• Loss of creativity. "Helpful assistant" tone flattens the output distribution.
• Capability degradation. Heavy preference optimization can cause forgetting of pretraining skills.
• Distribution shift. DPO trains on a fixed preference dataset; the policy moves away from it over time, reducing data relevance.

Mitigations: careful beta tuning, mixing SFT data into preference training, iterative online DPO (generate new pairs from the current policy), targeted rather than blanket safety training.

Check Your Understanding

Flashcards (click to flip)