How AI Models Are Trained: The Full Pipeline Explained

Training a large language model is not a single act — it is a five-stage industrial process that begins with petabytes of scraped text, passes through weeks of GPU computation consuming millions of dollars of electricity, and ends with a model that can hold a coherent conversation. Each stage has a specific job, a distinct failure mode, and a measurable cost. Here is how the whole pipeline works, from raw data to deployed product.

Stage 1: Data Collection and Cleaning

Every language model is, in a literal sense, a compressed statistical summary of its training data. The quality and composition of that data determine the model's capabilities more than almost any architectural choice.

Pretraining corpus — the large, diverse dataset used in the initial training phase, typically sourced from web crawls, books, code repositories, and curated sources such as Wikipedia. Common crawl snapshots alone exceed 250 terabytes (TB) of uncompressed text.

The raw web is full of noise: duplicate pages, machine-generated spam, adult content, personal data, and text so low in quality that training on it actively harms the model. A standard data pipeline runs several cleaning passes. Exact deduplication removes identical documents; near-deduplication (using MinHash algorithms) removes documents that are nearly identical. Language identification filters keep or discard text by language. Quality filters, often themselves small classifiers trained on human judgments of "good" versus "bad" text, score and threshold documents by perceived quality.

Data mixing is a deliberate decision: how much web text versus books versus code versus scientific papers? Research from Meta's LLaMA papers and Google's Gemini technical reports shows that the optimal mix is not obvious and varies with the target use case. A model trained with 20% code in pretraining performs substantially better on coding tasks than one trained on 5% code, even at the same total compute budget.

By the time data is ready for training, the raw internet crawl of several hundred terabytes has typically been filtered down to 10–50 terabytes of high-quality text for a mid-size model, or several hundred terabytes for a frontier model.

Source: Touvron et al. (2023), "Llama 2: Open Foundation and Fine-Tuned Chat Models," Meta AI — describes the data curation pipeline, deduplication strategies, and mixing ratios used for the LLaMA 2 family.

Source: Google DeepMind (2023), "Gemini: A Family of Highly Capable Multimodal Models," technical report — documents multimodal data sourcing and quality filtering approaches.

Stage 2: Tokenization

Before any text reaches a GPU, it must be converted into a sequence of integers. This conversion is done by a tokenizer — a component that splits text into subword units called tokens, then maps each token to an integer ID from a fixed vocabulary.

Token — an integer representing a chunk of text, typically a word, part of a word, or a punctuation character. The string "tokenization" might be split into the tokens ["token", "ization"] or ["t", "oken", "ization"], depending on the tokenizer's vocabulary.

The most common algorithm is Byte-Pair Encoding (BPE), which starts with individual characters, then iteratively merges the most frequent adjacent pairs until a target vocabulary size (typically 32,000 to 128,000 tokens) is reached. GPT-4 uses a 100,277-token vocabulary; LLaMA 3 uses 128,000.

Tokenization creates an important practical constraint: models have a context window — a maximum number of tokens they can process at once. Early GPT-3 had a 2,048-token context. Modern frontier models support 128,000 to over 1,000,000 tokens. Every token in the context costs memory and compute, so longer contexts are not free.

A useful analogy: tokenization is like reducing all possible English sentences to a fixed alphabet of about 50,000 syllable-like chunks, then treating each chunk as a single discrete unit. The model never sees letters — it sees integers, and the patterns it learns are statistical relationships between those integers.

Stage 3: Pretraining — Next-Token Prediction at Scale

Pretraining is the most expensive and most important stage. The model — almost always a transformer architecture with billions of parameters — is trained on the simple task of predicting the next token in a sequence, given all the tokens that came before.

Transformer — a neural network architecture introduced in 2017 that uses self-attention mechanisms to weigh the relevance of every token in a sequence against every other token. Self-attention is what allows a transformer to connect "it" in a sentence back to "the cat" ten words earlier.

The training objective is called cross-entropy loss: for each position in the training text, the model outputs a probability distribution over its entire vocabulary, and the loss measures how surprised the model was by the actual next token. Minimizing this loss over trillions of tokens forces the model to internalize grammar, facts, reasoning patterns, and world knowledge encoded in the training data.

This works because next-token prediction is a proxy for understanding. A model that can reliably predict "the patient was discharged from the ___" will have learned, implicitly, that hospitals discharge patients and that the blank should be "hospital." No explicit fact-injection occurs; the knowledge emerges from pattern learning.

The compute required scales with three factors: number of parameters, number of training tokens, and hardware efficiency. The Chinchilla scaling laws (Hoffmann et al., 2022) showed empirically that for a given compute budget, the optimal strategy is to train a smaller model on more tokens rather than a larger model on fewer tokens. A 70-billion-parameter model trained on 2 trillion tokens outperforms a 175-billion-parameter model trained on 300 billion tokens at the same compute cost.

Source: Hoffmann et al. (2022), "Training Compute-Optimal Large Language Models," DeepMind — the Chinchilla paper establishing that optimal token count should be approximately 20x the model parameter count.

What pretraining costs

Model size	Training tokens	Estimated compute cost (2025–2026)
7B parameters	1–2 trillion	$50,000–$500,000
70B parameters	2 trillion	$1.2M–$6M
Frontier model (>400B)	10–15 trillion	$50M–$150M

Training a frontier model like GPT-4 or Gemini Ultra is estimated to have cost $50–100 million in compute at 2023–2024 prices. H100 GPUs, at $7–$12 per GPU-hour in 2025 cloud pricing, are the workhorses of pretraining. A typical pretraining run uses 1,000–16,000 H100s simultaneously, with training wallclock time measured in weeks to months.

Source: Galileo AI (2025), "How Much Does LLM Training Cost?" — documents cloud GPU pricing and per-model cost estimates for pretraining and fine-tuning.

Stage 4: Supervised Fine-Tuning

A pretrained model is a powerful text predictor, but it has no concept of helpfulness, safety, or instruction-following. If you ask it "how do I reset my password?" it might continue the sentence in any direction — narrating a story, listing random software documentation, or generating a forum post that starts with that question.

Supervised fine-tuning (SFT) — a second training stage on a much smaller, curated dataset of (instruction, response) pairs, which teaches the model to produce outputs in the format and style expected by users.

SFT datasets are typically created by human annotators who write or curate thousands to tens of thousands of example conversations. The model is trained on these examples using the same next-token prediction objective as pretraining, but the data distribution is radically different: instead of diverse internet text, the model sees structured instruction-response pairs. This shifts the model's learned distribution from "predict the next word in a web document" to "produce a helpful response to an instruction."

SFT is far cheaper than pretraining — a fine-tuning run on a 7B model can be completed in hours on a single A100 GPU for $50–$500, depending on dataset size and target quality. The data, however, is expensive: high-quality annotated examples cost $5–$100 each to produce, and 10,000–100,000 examples are typically needed.

Stage 5: Reinforcement Learning from Human Feedback (RLHF)

SFT alone tends to produce models that are inconsistently helpful and can generate harmful content. RLHF is the stage that aligns the model more closely with human preferences.

RLHF — a training technique in which a separate reward model is trained on human preference judgments (annotator A rates response 1 higher than response 2), and the language model is then fine-tuned using reinforcement learning to maximize the reward model's score.

The process runs in three steps:

1. Collect preference data. Human annotators are shown two or more model outputs for the same prompt and rank them by quality, helpfulness, and safety. Large labs collect hundreds of thousands of such comparisons.
2. Train a reward model. A separate neural network — often a smaller version of the base model — is trained to predict human preference rankings. This reward model learns a scalar score for any (prompt, response) pair.
3. Fine-tune with RL. The language model is updated using a policy gradient algorithm (typically PPO — Proximal Policy Optimization) to generate outputs that score highly on the reward model, while a KL-divergence penalty prevents it from drifting too far from the SFT distribution.

A critical bottleneck: producing 600 high-quality RLHF annotations costs approximately $60,000 — about $100 per annotation — according to publicly documented figures from AI labs. For frontier models requiring millions of annotations, this cost can exceed the compute cost of pretraining.

RLAIF (Reinforcement Learning from AI Feedback) is a variant where another AI system (typically a capable frontier model) serves as the preference judge rather than a human annotator. Constitutional AI, developed by Anthropic, is an RLAIF variant where the AI critiques its own outputs against a set of principles. This approach dramatically reduces data costs but introduces dependence on the quality of the judge model.

Source: Ouyang et al. (2022), "Training language models to follow instructions with human feedback," OpenAI — the foundational RLHF paper describing the InstructGPT training process that underpins ChatGPT.

Source: Bai et al. (2022), "Constitutional AI: Harmlessness from AI Feedback," Anthropic — describes RLAIF and Constitutional AI as an alternative to pure human-preference labeling.

The 2025–2026 alignment stack

The post-training landscape has grown more sophisticated since the original InstructGPT RLHF paper. By 2026, major labs use modular pipelines that combine SFT, preference optimization (DPO, SimPO, KTO — algorithms that train directly on preference data without a separate reward model), and reinforcement learning with verifiable rewards (GRPO, DAPO) for tasks like mathematics and code where correctness can be checked automatically.

Method	Reward signal	Cost	Best for
SFT	None (supervised)	Low	Instruction format, style
RLHF (PPO)	Human preferences	High	Open-ended quality
DPO	Human preferences (direct)	Medium	Preference alignment without RL complexity
RLAIF	AI judge preferences	Medium-low	Scale, cost reduction
RLVR (GRPO/DAPO)	Verifiable correctness	Low	Math, code, logic

Stage 6: Evaluation

A trained model is only as trustworthy as its evaluation suite. Evaluation in 2026 runs on three parallel tracks.

Benchmark evaluation tests the model on standardized tasks: MMLU (57-subject multiple-choice knowledge), HumanEval (code generation), MATH (competition mathematics), HellaSwag and WinoGrande (commonsense reasoning). Benchmarks have a known limitation: models can be, and sometimes are, inadvertently or deliberately trained to score well on specific benchmarks without improving real-world capability.

Human evaluation uses annotators to rate model outputs on helpfulness, harmlessness, and honesty — often called the HHH framework. Crowdsourced evaluation platforms like Chatbot Arena (LMSYS) allow public users to compare models without knowing which model produced which output, providing naturalistic preference signal.

Red-teaming attempts to elicit harmful, biased, or incorrect outputs through adversarial prompting. Major labs run red-teaming exercises before public releases, and some also fund external red teams.

Source: Hendrycks et al. (2021), "Measuring Massive Multitask Language Understanding," UC Berkeley — the MMLU benchmark paper, widely used for evaluating model knowledge across domains.

The Compute Infrastructure Behind Training

Pretraining a frontier model requires not just raw GPU count but also high-speed interconnects between GPUs. A single transformer training run cannot fit on one GPU — the model is split across thousands of GPUs using data parallelism (each GPU processes a different batch), tensor parallelism (the model's weight matrices are split across GPUs within a layer), and pipeline parallelism (layers are split across GPUs sequentially).

FLOP (floating-point operation) — a single arithmetic computation. Training GPT-3 (175B parameters) on 300 billion tokens required approximately 3.14 × 10²³ FLOPs. An H100 GPU performs approximately 1,000 trillion FLOPs per second (1 petaFLOP/s) in mixed precision.

NVIDIA H100 GPUs dominate frontier model training in 2025–2026. A single H100 80GB costs $25,000–$35,000 new; cloud rental runs $7–$12 per GPU-hour. Meta's Llama 3 training cluster used 24,576 H100 GPUs. Google TPU v5p pods are an alternative for Google-internal training.

The energy footprint is significant: training a single large model consumes between 100 and 1,000 megawatt-hours (MWh) of electricity, roughly equivalent to the annual electricity consumption of 10–100 US households.

Source: Strubell et al. (2019), "Energy and Policy Considerations for Deep Learning in NLP," University of Massachusetts Amherst — the canonical study on energy costs of NLP model training, updated estimates appear in subsequent literature.

What This Means

Training a large AI model is an industrial-scale engineering problem as much as a research one. The core learning algorithm — predict the next token, update weights, repeat — is conceptually simple, but the execution involves petabyte-scale data pipelines, thousands of synchronized GPUs, and multi-stage alignment processes that cost as much as building a commercial aircraft. Understanding this pipeline clarifies both the capabilities and the limitations of modern AI: the model knows what its data contained, can follow instructions it was fine-tuned on, and reflects the preferences of the humans who labeled its RLHF data. None of that is magic — it is engineering with known failure modes and measurable costs.

Why Training Runs Fail — and What Engineers Watch For

A multi-week pretraining run is not a set-and-forget process. A 70-billion-parameter model trained across 4,000 H100 GPUs on 2 trillion tokens is a distributed system with thousands of simultaneous failure modes, and the engineers running it monitor a live dashboard of metrics throughout. Understanding those failure modes helps explain both why frontier model training is so expensive and why checkpointing and run management are treated as first-order engineering concerns.

Loss spikes and training divergence are the most common catastrophic failure. Training loss — the cross-entropy measure of how surprised the model is by the next token — should decline smoothly over the course of a run, with small oscillations. A loss spike is a sudden upward jump, sometimes of 0.5 to several units, that indicates the model's weights moved to an unusually bad region of the loss landscape. Mild spikes self-recover. Severe spikes (sometimes called "loss explosions") represent actual divergence: the model's outputs have become incoherent, and the run is essentially dead. Engineers first encountered these at scale in the early days of large model training and they remain a routine concern. The standard defensive mechanism is aggressive checkpointing — saving the full model state every few hundred training steps — so that when a spike occurs, the run can be rolled back to the last stable checkpoint rather than restarted from scratch.

Learning-rate schedules are one of the primary levers for preventing divergence. The learning rate controls how large each parameter update step is. A learning rate that is too high on a given batch causes updates that overshoot the local loss minimum, producing instability. A learning rate too low makes training prohibitively slow and can cause the model to get stuck in poor regions. Standard practice is to use a warmup phase — starting with a low learning rate and ramping up over the first few thousand steps — followed by a cosine decay schedule that gradually reduces the learning rate toward the end of training. The warmup prevents the large random gradients early in training from producing runaway updates before the model has learned any useful representations. Getting the learning rate schedule wrong for a given batch size and model size is among the most common causes of underperforming runs.

Source: Hoffmann et al. (2022), "Training Compute-Optimal Large Language Models," DeepMind — documents the relationship between batch size, learning rate, and optimal training token count, with hyperparameter sensitivity analysis.

Gradient instability at scale is a distinct problem from learning rate misconfiguration. In very large models, gradients can become extremely large (gradient explosion) or extremely small (gradient vanishing) due to the depth of the network and the multiplicative nature of backpropagation through many layers. The standard countermeasure is gradient clipping — if the global gradient norm exceeds a threshold (typically 1.0), all gradients are rescaled proportionally to bring the norm back within bounds. This prevents individual bad batches from producing destructive weight updates. Every major training framework (PyTorch FSDP, JAX/XLA, Megatron-LM) applies gradient clipping by default in large-model training.

Data contamination is a subtler failure mode that is not always detected during a run. Contamination occurs when evaluation benchmark data — the questions and answers the model will be tested on — are present in the training corpus. A contaminated model can score well on benchmarks without having genuinely learned the underlying capability, misleading the development team about the model's quality. Detection requires careful deduplication between training data and evaluation sets, which is non-trivial given that benchmark data can appear in web crawls through forum discussions, study guides, and reproduced text. Google's Gemini technical report and Meta's LLaMA 3 report both describe explicit deduplication steps against common benchmarks, acknowledging this as a real risk.

What engineers actually watch on the monitoring dashboard during a multi-week pretraining run includes: training loss (smoothed moving average and raw), validation loss on held-out data (a separate sample not seen during training, used to detect overfitting), gradient norm (watching for spikes), GPU utilization (watching for hardware failures causing idle GPUs), memory usage (approaching OOM — out-of-memory — errors), and temperature metrics for the GPU cluster (thermal throttling reduces effective throughput). At Meta, Google, OpenAI, and Anthropic, dedicated teams run 24-hour monitoring during active pretraining runs because hardware failures in a cluster of thousands of GPUs are not rare events — they are expected. A single GPU dropping offline in the middle of a forward pass needs to be detected and routed around without restarting the entire run.

Hardware failures and fault tolerance are addressed through distributed checkpointing and automatic job rescheduling. In a cluster of 8,000 GPUs, individual GPU failures may happen every few days. Training frameworks are designed to detect these, checkpoint the current model state, restart the failed node with a replacement GPU, and resume from the last checkpoint — ideally without human intervention and with minimal training time lost. Meta's Llama 3 training infrastructure paper described automatic fault detection and recovery as a significant engineering investment, noting that the effective GPU utilization (time actually spent on training vs. recovery and restart) is a key efficiency metric that distinguishes mature training infrastructures from newer ones.

Source: Meta AI (2024), "The Llama 3 Herd of Models," technical report — details the training infrastructure, fault tolerance mechanisms, and monitoring practices used for the Llama 3 70B and 405B training runs.

Frequently Asked Questions

Q: How long does it take to train a large language model?
A: Pretraining a frontier model (400B+ parameters) on 10 trillion tokens typically takes 3–6 months of continuous GPU cluster operation. Smaller models (7B parameters) can be pretrained in days to weeks. Fine-tuning and RLHF add weeks to the timeline for large models.

Q: Why can't you just train on more data indefinitely?
A: The Chinchilla scaling laws show diminishing returns from adding data beyond the compute-optimal ratio of roughly 20 tokens per parameter. Beyond data quantity, running out of high-quality data is a real constraint — estimates suggest the internet's stock of high-quality English text may be fully utilized by frontier labs by 2026–2028.

Q: What is the difference between pretraining and fine-tuning?
A: Pretraining is training on a massive, diverse corpus to give the model broad world knowledge, using billions of dollars of compute. Fine-tuning is a shorter, cheaper second stage on a smaller, curated dataset that shapes how the model responds to users. You need pretraining first; fine-tuning alone on a randomly initialized model produces nothing useful.

Q: Does RLHF make models safer?
A: RLHF reduces specific harmful behaviors that annotators flagged during preference labeling, but it does not guarantee safety in a general sense. Models can still be jailbroken, can still produce confident misinformation, and can reflect the biases of their annotators. RLHF is alignment, not a safety proof.

Q: What is a parameter in a neural network?
A: A parameter is a numerical weight inside the model — a single floating-point number that gets adjusted during training. A 70-billion-parameter model has 70 billion such numbers. At 2 bytes each (bfloat16 precision), storing 70B parameters requires 140 GB of memory, which is why large models need multiple GPUs.

Q: Can small companies train frontier models?
A: The compute and data costs for frontier models (GPT-4 class, Gemini Ultra class) are effectively out of reach for organizations without hundreds of millions of dollars of dedicated infrastructure. However, the open-weight ecosystem (LLaMA 3, Mistral, Qwen, Falcon) means smaller organizations can fine-tune already-pretrained models at costs ranging from hundreds to tens of thousands of dollars.

---

Read next:

• Why AI Models Give Different Answers to the Same Question
• How Large Language Models Work, Explained
• What Is Machine Learning?