From pattern machine to helpful assistant

The Completion Trap: Base vs Aligned

Let's see the difference concretely. Below, try different prompts and compare how a base model and an instruction-tuned model respond. Notice that the base model isn't "broken" — it's doing exactly what it was trained to do: continue text.

Chat Templates: Teaching Turn Boundaries

Before we even touch weights, we can partially bridge the gap with chat templates. These use special tokens — unique IDs in the vocabulary that never appear in normal text — as structural signals.

<bos><start_of_turn>user
Hello, how are you?<end_of_turn>
<start_of_turn>model
I'm doing great. How can I help?<end_of_turn>

Tokens like <end_of_turn> serve as stop signs. During post-training, the model learns: "when I emit <end_of_turn>, I'm done speaking — the other side gets a turn." The application code watches for this token and stops sampling.

Every model family uses different special tokens (ChatML, Llama, Gemma, Mistral — all different). HuggingFace hides this behind tokenizer.apply_chat_template():

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-it")
messages = [
    {"role": "user", "content": "Hello, how are you?"},
    {"role": "assistant", "content": "I'm doing great. How can I help?"}
]
formatted = tokenizer.apply_chat_template(messages, tokenize=False)

The Post-Training Pipeline

Modern post-training has two main stages, sometimes followed by tools and safety work:

Stage 1 (SFT) makes the model capable — it learns to respond at all. Stage 2 (Preference Optimization) makes it aligned — it learns which responses are better. A model that only received SFT is helpful but might happily explain how to synthesize VX nerve gas. Preference optimization tells it "no, that one's worse".

Full Fine-Tuning: The Naive Approach

The most straightforward way to do SFT: just keep training. Take all the model's parameters and run backpropagation on instruction data. Mechanically identical to pretraining — same forward pass, same cross-entropy loss, same AdamW optimizer. Only the data is different.

from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

model = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-pt")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-pt")
dataset = load_dataset("timdettmers/openassistant-guanaco")

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset["train"],
    args=SFTConfig(output_dir="./results", max_seq_length=512),
)
trainer.train()

LoRA: The Sticky-Note Trick

Insight from Hu et al. 2021: the weight changes needed during fine-tuning are low-rank. You don't need to modify every dimension of every weight matrix — the useful update lives in a small subspace.

So instead of learning the full delta matrix ΔW (the same shape as W), we decompose it:

W_new = W_frozen + ΔW
ΔW ≈ A × B     where A: (d × r),  B: (r × d),  r ≪ d

Concretely, for one 4096 × 4096 attention projection:

At inference time, LoRA adapters can be merged back into the base weights (W_merged = W + A·B), so there's zero runtime overhead. You can also keep them separate and hot-swap different adapters for different personas or domains.

The Alignment Problem

Why is all this machinery necessary? Because the pretraining objective — "predict the next token on the internet" — is not the same thing as "be helpful to a specific human asking a specific question". This gap is called the alignment problem.

Since we can't directly optimize "be helpful", post-training uses a clever two-step dodge:

1. SFT: show the model examples of helpful behavior (curated responses). It learns to imitate them.
2. Preference optimization: show the model pairs of responses and tell it which one humans prefer. It learns to produce the preferred style even on prompts it has never seen.

This works because both steps use the one thing we can compute: the log-probability the model assigns to any given text. SFT increases the log-prob of good responses. Preference optimization increases the log-prob of chosen responses relative to rejected ones.

Check Your Understanding

Flashcards (click to flip)