Tulu 3: The most transparent post-training recipe ever released

Ai2's Tulu 3 (November 2024) is the post-training equivalent of LLaMA 3 -- a fully documented, fully open end-to-end post-training pipeline. Every dataset, every training script, every intermediate checkpoint, every evaluation is released. It built on LLaMA 3.1 base models and matched GPT-4o-mini on most benchmarks.

OLMo 3 (November 2025) takes it further: a fully open pipeline including the base model (trained on Dolma 3, 9.3T tokens). Both are the clearest public window into modern post-training.

Design Philosophy: Transparency, Systematic Experimentation, Verifiable Improvement

Three core principles:

1. Transparency. Release everything -- not just weights, but all training data, code, and intermediate checkpoints.
2. Systematic experimentation. Controlled ablation studies instead of intuition-driven tweaks.
3. Verifiable improvement. Rigorous evaluation with development/unseen benchmark splits to prevent implicit overfitting.

For years, open post-training lagged behind proprietary methods. Tulu 3 closed that gap -- not with a secret technique but with clean, systematic execution.

The Four-Stage Pipeline

Stage 1: Data Curation (939,344 prompts)

• Public datasets (57%): WildChat, OpenAssistant, No Robots, FLAN v2, OpenMathInstruct, NuminaMath, Evol-CodeAlpaca, Aya, SciRIFF, TableGPT, Daring-Anteater.
• Synthetic data (43%): generated using a persona-driven methodology. ~150K math word problems across difficulty levels, programming challenges across languages, instruction-following tasks with verifiable constraints.

Response generation: for prompts without high-quality responses, they used GPT-4o or Claude 3.5 Sonnet to generate new ones, then filtered out empty responses, model self-references, and weak outputs.

Decontamination: 8-gram matching against evaluation benchmarks. Critical finding: 11.3% of NuminaMath-TIR overlapped with MATH evaluation problems. Any matching examples were removed. Without this step, benchmark scores would have been inflated.

Stages 2-3: SFT and Preference Optimization

SFT: Iterative Data Mixing

The innovation: iterative mixing, not one-shot training. Process:

1. Compare current model against SOTA on each target skill
2. Create specialized models (math-only, code-only) to establish upper bounds
3. Merge successful mixtures, add/remove datasets to fix lagging skills
4. Remove evaluation overlaps and reduce oversized datasets

Ablation findings: synthetic data boosts math and code; removing any single capability dataset degrades that skill without helping others; data mixing ratios matter more than total volume; response quality matters more than prompt quantity.

DPO: On-Policy Preference Data

~300K prompts with preference pairs (5x larger than UltraFeedback's 60K). Process:

1. Collect diverse prompts
2. Generate multiple responses per prompt using multiple models, including the Tulu 3 SFT model itself
3. Score with reward models
4. Select chosen/rejected pairs

On-policy vs off-policy: including responses from the Tulu 3 SFT model (on-policy) improved performance compared to only external model responses (off-policy). The model learns better from its own mistakes.

Algorithm selection: DPO, PPO, and SimPO all performed comparably with proper tuning. Algorithm choice matters less than data quality. Length normalization is critical to prevent rewarding verbosity.

Stage 4: RLVR (Reinforcement Learning with Verifiable Rewards)

Replace the neural reward model with a verification function for tasks with objectively correct answers:

def compute_rlvr_reward(prompt, response, task_type):
    if task_type == "math":
        # Extract boxed answer, compare to ground truth
        return 1.0 if correct else 0.0
    elif task_type == "code":
        # Run unit tests
        return fraction_passed
    elif task_type == "format":
        # Check that response follows required structure
        return 0.1 if format_matches else 0.0

Benefits: no reward model needed, no reward hacking possible, no human annotation bias.

Training domains: math (exact answer verification) and code (unit test execution).

Infrastructure for the 405B model: 256 GPUs total, 16 for inference (vLLM with tensor parallelism) and 240 for gradient updates. The split matters because RL needs both generation and training to run concurrently.

Results: RLVR provides targeted improvements (math, code) without degrading general capabilities. Key finding: RLVR improves more at larger scales -- the 405B model showed larger gains than 8B or 70B. As base models improve, returns to RL-based post-training may increase.

Safety, Evaluation, and OLMo 3

Safety throughout the pipeline, not as an afterthought. Three key datasets: harmful request examples, refusal demonstrations, and contrastive benign examples (from WildJailbreak and CoCoNot).

Key finding: safety is orthogonal to capability. Removing safety data affects safety metrics but NOT general benchmarks. This means there is no fundamental trade-off between being safe and being capable -- safety must be explicitly trained but can be added without hurting performance.

Evaluation innovation: dev vs unseen splits. Development benchmarks are used during training to tune data mixtures. Unseen benchmarks are held out entirely and only reported at the end. Prevents implicit benchmark overfitting.

OLMES (Open Language Model Evaluation System) standardizes prompting formats, shot counts, decoding parameters, and evaluation templates. Makes results reproducible across teams.

OLMo 3: Fully Open Alternative

OLMo 3 goes further than Tulu 3 by opening the base model too:

• Pretraining on Dolma 3 (9.3T tokens, ~5.9T used for final)
• Midtraining on Dolmino (100B tokens, math/science/code)
• Long context extension on Longmino (50B tokens)
• Post-training with Dolci mixes (SFT + DPO + RLVR)

Three variants: OLMo 3-Instruct (standard), OLMo 3-Think (chain-of-thought reasoning; 32B hits 96.2% on MATH), OLMo 3-RL Zero (RL directly on the base model, no SFT, releasing checkpoints for math, code, instruction following, and combined variants).

Efficiency improvements: 8x throughput for SFT, 4x for RL via in-flight weight updates and continuous batching.

Lessons and Check Your Understanding

1. Data curation matters most: systematic prompt collection and decontamination are critical.
2. On-policy preference data beats off-policy: the model learns better from its own generations.
3. Algorithm choice matters less than data quality: DPO, PPO, SimPO all comparable with proper tuning.
4. RLVR is the new frontier: verifiable rewards provide targeted improvements without capability degradation.
5. Safety is orthogonal to capability: can be added without hurting performance.
6. Decontamination is essential: 11.3% overlap in popular datasets silently inflates benchmarks.
7. Evaluation must be rigorous: dev/unseen splits prevent implicit overfitting.
8. Full transparency enables science: releasing everything lets the community verify and build upon results.

Flashcards (click to flip)