LLaMA 3: The most documented frontier training run in history

Meta's LLaMA 3 technical report is 92 pages and documents nearly every decision of a 16,384-GPU training run that produced the 8B, 70B, and 405B models. This module uses it as a capstone to revisit every pretraining concept from Modules 10-16.

The Design Philosophy: Data, Scale, Simplicity

Meta identified three levers:

1. Data quality and diversity would drive the biggest gains, more than architectural innovation.
2. Scale beyond previous LLaMA models (405B vs LLaMA 2's 70B).
3. Managing complexity -- consistently pick simpler, proven approaches over novel ones.

Biggest conservative choice: stick with a dense transformer rather than Mixture-of-Experts (MoE). DeepSeek V3 later achieved SOTA with ~10x less compute using MoE. But MoE adds routing complexity, load balancing issues, and harder-to-debug training dynamics. On 16,000 GPUs, stability matters more than efficiency.

Scaling Law Decisions and Overtraining by Design

Meta used a two-step scaling law approach:

1. Predict loss from compute using small-scale experiments (standard Chinchilla).
2. Map predicted loss to benchmark accuracy via a sigmoid -- because benchmark scores follow an S-curve, not a linear curve.

This predicted that a 402B model trained on 16.5T tokens would be compute-optimal for their budget. But they deliberately overtrained the 8B model -- it saw ~15T tokens (~1,875 tokens per parameter, nearly 100x Chinchilla). Reason: inference cost matters, and a well-trained small model is the sweet spot for deployment.

Data Engineering: The Bootstrap Approach

Meta used LLaMA 2 to help build LLaMA 3's training data:

• LLaMA 2 trained quality classifiers for web data
• LLaMA 3 checkpoints (mid-training) evaluated the value of new datasets
• The final LLaMA 3 generated synthetic data for post-training

Better models -> better data -> even better models. The final mix was approximately:

• 50% general knowledge
• 25% math and reasoning
• 17% code
• 8% multilingual

The mix was not static. Meta adjusted during training: more math later for reasoning, more recent web data to push knowledge cutoff forward, downsampled weaker subsets identified by quality metrics.

Interactive: LLaMA 3 Architecture and Parallelism

4D parallelism on the 405B model:

• Tensor Parallelism TP=8 within each NVSwitch-connected node
• Pipeline Parallelism PP=16, 126 layers split into 16 stages
• Data Parallelism via FSDP sharding weights/grads/optimizer
• Context Parallelism CP=16 (only enabled for long-context phase)

TP=8 x PP=16 = 128 GPUs to hold one copy of the model. Sustained 400 TFLOPs/GPU is ~40% of theoretical peak -- excellent for distributed training.

The Training Recipe: Three Phases

Evaluation, Fault Tolerance, Small-Model-First

Evaluation innovation: Meta used scaling laws not just to predict loss but to predict benchmark performance, extrapolated over four orders of magnitude of compute. Their predictions slightly underestimated final performance, which was a pleasant surprise.

Fault tolerance: 466 interruptions in 54 days, ~78% hardware-related. Meta achieved 90%+ effective training time through automated failure detection, redundant checkpoint storage (local SSD + network filesystem + cloud), and silent data corruption detection.

Small-model-first pattern: scaling law experiments, data mix experiments, and annealing experiments all ran on 8B models before committing to 405B scale. A bad decision on 8B costs thousands. The same decision on 405B costs millions.

Secondary use of annealing: Meta discovered they could use annealing runs to quickly evaluate new datasets. Take a 50% trained 8B model, anneal on 70% default + 30% new data, compare benchmark changes. Not as rigorous as full scaling experiments but much cheaper for early screening.

Check Your Understanding

Flashcards (click to flip)