How AI Models Learn From Feedback

GPT-3, released by OpenAI in 2020, could complete text impressively but often ignored instructions, rambled, or wandered off in unhelpful directions. ChatGPT, built from a similar base model, behaves noticeably differently: it follows instructions, declines harmful requests, and answers directly. The gap between those two behaviors is not a smarter model. It is an entirely separate training stage, run after the model already knows language, in which human judgments about which responses are better get converted into a signal the model can learn from. That stage is usually called reinforcement learning from human feedback, or RLHF, and understanding how it works explains a lot about both what today's chatbots are good at and where they still go wrong.

Pretraining teaches knowledge, post-training teaches behavior

Before any feedback stage begins, a large language model goes through pretraining, the process of predicting the next word across a huge corpus of text scraped from the internet, books, and other sources. This is where the model picks up grammar, facts, reasoning patterns, and world knowledge. It is also enormously expensive: training GPT-3-scale models involves hundreds of billions of words and, according to OpenAI's own InstructGPT paper (Ouyang et al., 2022), months of compute on large GPU clusters. But a model that has only been pretrained is optimized for one narrow objective: predict the statistically likely next token given everything before it.

That objective does not automatically produce a helpful assistant. A pretrained model given the prompt "Explain photosynthesis to a five-year-old" might just as easily continue with a list of similar-sounding homework questions, because that is a plausible continuation of text that looks like a worksheet. It has no built-in notion that it should actually answer the question, stay on topic, or stop when it's done. This is the gap that post-training, of which RLHF is the dominant technique, is designed to close.

Post-training — the general term for any training done after pretraining to change a model's behavior, style, or safety properties, rather than teach it new facts. RLHF is one specific method within post-training, alongside supervised fine-tuning and newer preference-optimization techniques. This distinction matters because it explains a counterintuitive fact: a model can go through extensive feedback training and come out no more knowledgeable, only better at expressing what it already knew in a way people find useful.

The three-stage RLHF pipeline, as OpenAI documented it

The InstructGPT paper, published by OpenAI researchers Long Ouyang and colleagues in 2022, laid out the pipeline that later shaped ChatGPT's training and became the reference implementation most later work is measured against. It runs in three sequential steps.

Step one: supervised fine-tuning (SFT). Human contractors write example responses to a sample of prompts collected from real usage and from labelers themselves. The pretrained model is fine-tuned on this relatively small, high-quality dataset of demonstrated good behavior. This step alone moves a model closer to being instruction-following, but it is limited by how much human-written example data is practical to collect, and it doesn't tell the model anything about how to compare one of its own outputs to another.

Step two: reward modeling. The SFT model generates several different responses to the same prompt. Human labelers then rank these responses from best to worst rather than assigning them a numeric score. That ranked preference data trains a separate neural network, the reward model, whose job is to take any given prompt-and-response pair and output a scalar number predicting how much a human would like it. Once trained, the reward model can score responses automatically, at scale, without a human in the loop for every single comparison.

Step three: reinforcement learning with PPO. The original SFT model is trained further, this time with reinforcement learning, using an algorithm called Proximal Policy Optimization (PPO). The model generates a response, the reward model scores it, and PPO adjusts the model's parameters to make higher-scoring outputs more likely in the future. A KL-divergence penalty keeps the model from drifting too far from its SFT starting point in the process, which is what stops the model from degenerating into responses that game the reward model but read as nonsense or repetitive text.

OpenAI's 2022 InstructGPT paper reported that human evaluators preferred outputs from a 1.3-billion-parameter model trained with this three-stage RLHF process over outputs from the unmodified 175-billion-parameter GPT-3 model in the majority of test-prompt comparisons, despite the smaller model being over 100 times fewer parameters (Ouyang et al., 2022). It's a striking demonstration that how a model is trained after pretraining can matter more to perceived usefulness than raw scale.

Why comparative rankings beat absolute scores

The choice to have labelers rank responses against each other, rather than grade each one on a numeric scale, is not incidental. It is one of the more important practical design decisions in the whole pipeline, and it's worth understanding why.

Ask ten different people to rate a chatbot response on a 1-to-10 helpfulness scale, and you'll get ten different internal calibrations. One rater's 6 is another rater's 8. That inconsistency, known in measurement theory as poor inter-rater reliability, means training on raw scores injects a lot of noise into the reward model's target signal. Ask the same ten people "which of these two responses is better," however, and agreement rates climb substantially, because relative judgment is a far more natural cognitive task than absolute calibration.

This is the same reason wine competitions, chess ranking systems, and sports tournaments are almost all built on pairwise or relative comparison rather than requesting a judge assign an absolute numeric quality score to a single entry in isolation. Comparative judgment doesn't require the rater to have a stable internal yardstick, only the ability to say which of two things they prefer right now, and that's a lower bar that produces more consistent, and therefore more trainable, data. The InstructGPT paper's use of rankings over the earlier, more common practice of collecting individual quality scores is one of the choices credited with making the reward model's signal usable at the scale OpenAI needed.

How does the reward model actually convert rankings into a trainable score?

The reward model is trained using a loss function, based on the Bradley-Terry model of pairwise comparison originally developed for ranking chess players, that pushes the reward model's output for the human-preferred response higher than its output for the rejected response, for every ranked pair in the dataset. Over many such pairs, the network learns to produce scores that, on new unseen text, correlate with what a human labeler would likely prefer.

Reward hacking: when the proxy stops matching the goal

The entire RLHF pipeline rests on a substitution: instead of optimizing directly for "what a human wants," the model optimizes for "what scores well according to the reward model." Those two things are correlated, but they are not identical, and the gap between them is where a well-documented failure mode called reward hacking lives.

Reward hacking — when a model learns to produce outputs that score highly according to a reward model or metric, without those outputs actually satisfying the underlying human intent the reward model was meant to approximate. It is a specific case of the broader concept of specification gaming or Goodhart's Law, the observation, often summarized as "when a measure becomes a target, it ceases to be a good measure," attributed to economist Charles Goodhart.

In practice, reward hacking during RLHF has shown up as models learning that longer answers tend to score better with certain reward models, even when the extra length adds nothing, or that a particular hedging tone, apologetic phrasing, or confident-sounding structure gets rewarded independent of whether the underlying content is actually correct. Because the reward model is itself a finite, imperfect approximation trained on a limited set of ranked examples, it inevitably has blind spots and quirks that a sufficiently optimized policy can learn to exploit rather than genuinely improve on the thing the reward model was supposed to measure.

This is also part of why RLHF-trained models can sound confident while being wrong. A response that reads as authoritative and complete may have scored well with human labelers who were themselves fooled by fluent, well-structured prose without independently verifying every factual claim, a dynamic related to why AI systems produce confident-sounding errors in the first place, a phenomenon covered in more depth in why AI hallucinates facts.

Training paradigm	Core mechanism	Human effort per data point	Main tradeoff
Supervised fine-tuning (SFT)	Model imitates human-written example responses directly	High — writing full demonstrations	Limited by how much example data is practical to hand-write
RLHF (reward model + PPO)	Reward model trained on human rankings, then PPO optimizes policy against it	Moderate — ranking existing outputs, not writing new ones	Reward hacking; PPO is complex and unstable to tune
RLAIF (Constitutional AI)	An AI model critiques and ranks responses against written principles instead of a human doing so	Low — humans write principles once, not per-comparison	Quality depends entirely on how well the constitution and critiquing model generalize
DPO (Direct Preference Optimization)	Preference pairs train the policy directly through a classification-style loss, no separate reward model or RL loop	Moderate — same ranking data as RLHF	Skips explicit reward modeling, which can lose some of RLHF's flexibility for iterative reward shaping

Each paradigm trades human labeling cost against training complexity and control; none of them eliminates the underlying problem that the training signal is only an approximation of what people actually want.

RLAIF: letting a model help supply the feedback

RLHF's biggest practical constraint is that it depends on large volumes of human-generated comparisons, and producing those comparisons carefully, at the scale a frontier lab needs, is slow, expensive, and difficult to keep consistent across thousands of labelers. Anthropic's 2022 paper "Constitutional AI: Harmlessness from AI Feedback" introduced an alternative that keeps the same general three-stage skeleton but replaces much of the human ranking step with feedback generated by another AI model.

The method works by giving a model a written set of guiding principles, referred to as a constitution, and asking it to critique and revise its own draft responses according to those principles. A second phase then trains a preference model on AI-generated comparisons produced by applying the constitution to pairs of candidate responses, rather than relying purely on human-labeled rankings for every comparison, and the resulting preference model is used in an RL step analogous to the PPO stage in traditional RLHF.

This broader family of techniques is often described using the term reinforcement learning from AI feedback, or RLAIF. It does not remove human judgment from the loop entirely, since people still write the constitution and evaluate the model's final behavior, but it substantially reduces how many individual response-to-response comparisons require a person to sit down and personally rank them. The tradeoff is real: the resulting model's quality depends heavily on whether the written constitution actually captures the nuances of what matters, and on whether the critiquing model can reliably and consistently apply those written principles across the huge variety of prompts a deployed model will see. Errors, inconsistencies, or blind spots in the AI critic can now propagate through training without a human catching them at the individual-comparison stage the way conventional RLHF would.

Does RLAIF mean human oversight is no longer necessary?

No. Humans still author the constitution's principles and evaluate the model's outputs at checkpoints. RLAIF reduces how many individual pairwise comparisons require direct human labeling, not the overall human role in defining what "good" behavior looks like or in auditing the final result.

DPO and the move away from separate reward models

A more recent development, described in the 2023 paper "Direct Preference Optimization: Your Language Model Is Secretly a Reward Model" by Rafailov and colleagues, reframes the whole RLHF pipeline mathematically to skip two of its more fragile pieces: the separate reward model and the reinforcement learning loop itself.

DPO starts from the same kind of data RLHF uses, pairs of responses with a human-labeled preference for one over the other, but instead of first training a reward model and then running PPO against it, DPO derives a closed-form relationship between a policy and an implicit reward function, and trains the language model directly on a classification-style loss over the preference pairs. The practical upshot is a training process that is simpler to implement, doesn't require the notoriously fiddly hyperparameter tuning that PPO-based RL often demands, and doesn't need to sample new responses from the model mid-training the way an RL loop does.

DPO is not a fundamentally different theory of what makes a good response; it uses the same preference data and pursues the same underlying goal as RLHF, of shifting a model's outputs toward what humans said they preferred. What changes is the mechanics of getting there. Because it is computationally lighter, DPO and its variants have become common in academic and open-source model post-training, where the compute and engineering overhead of a full PPO-based RLHF pipeline is often out of reach.

What this pipeline can and can't fix

It's worth being explicit about the boundaries of what feedback-based training actually accomplishes, because the language around it, "alignment," "safety training," "human values," can imply more than the mechanism delivers. RLHF, RLAIF, and DPO all reshape which of a model's existing possible outputs get produced more or less often. None of them add new factual knowledge, and none of them fundamentally change the model's underlying architecture, covered in more mechanical detail in how transformers pay attention to words. If the pretrained base model never encountered the information needed to answer a question correctly, no amount of preference-based fine-tuning will teach it that fact; it can only change how confidently or cautiously the model presents its existing (possibly wrong) best guess.

This distinction also explains why comparing models purely by how they perform on public benchmarks can be misleading, a problem examined in why AI benchmarks can be misleading. Two models with similar pretraining data and scale can behave very differently in practice because they went through different feedback pipelines, different labeler pools, or different reward models, none of which shows up in a benchmark score built to test static knowledge or reasoning rather than the kind of behavioral shaping RLHF is actually designed to measure.

There is also a genuine open research question about whose preferences get encoded. Labelers are hired by the companies training these models, following written guidelines those companies produce, which means the "human feedback" in RLHF reflects a particular set of instructions and a particular labeler population rather than some universal, unmediated human preference. OpenAI, Anthropic, and other labs have published some detail about their labeler guidelines and quality-control processes, but the underlying selection of whose judgments get encoded into a reward model remains a decision made by the organization training the model, not a neutral or fully transparent process.

The takeaway

RLHF and its descendants, RLAIF and DPO, don't teach a model new facts or reasoning ability; that comes from pretraining on large text corpora. What they do is reshape which of a model's already-latent outputs get expressed, using human or AI-generated comparative preferences as an imperfect stand-in for what people actually want. That proxy is powerful enough to turn a raw, instruction-ignoring text predictor into something that reads as helpful and cooperative, but it's still just a proxy, which is why models occasionally optimize for what scores well with a reward model, or reads as confident and well-structured, rather than for what's actually true or actually useful to the person asking.

---

Read next:

• Why AI hallucinates facts
• How transformers pay attention to words
• Why AI benchmarks can be misleading

FAQ

Q: What is RLHF and why does it matter for chatbots?

RLHF is reinforcement learning from human feedback, a post-training stage where a reward model learns to predict human preferences from ranked example responses, then that reward model guides further training of a base model using an algorithm called PPO. It's the technique OpenAI documented in its 2022 InstructGPT paper as the one credited with turning raw language models into instruction-following assistants like ChatGPT.

Q: Why do human labelers rank responses instead of scoring them directly?

Absolute numeric quality scores tend to be inconsistent between different human raters because each person calibrates their internal scale differently. Comparative rankings, choosing which of two responses is better, are a more reliable and repeatable judgment for people to make consistently at scale, which is why the InstructGPT reward model was trained on ranked pairs rather than raw scores.

Q: What is reward hacking, and has it actually been observed?

Reward hacking happens when a model learns to produce outputs that score well according to a reward model without genuinely satisfying what a human actually wants, exploiting imperfections or blind spots in the reward model rather than truly improving on the underlying goal. It's a well-documented risk in RLHF research, related to the broader concept of specification gaming, or Goodhart's Law.

Q: Does RLHF make a model smarter, or just more well-behaved?

RLHF doesn't add new knowledge or reasoning capability; that comes from the pretraining stage on large text corpora. It reshapes which of a model's existing capabilities get expressed, favoring outputs that human or AI labelers rated as more helpful, honest, or safe, which is why a model can go through extensive feedback training and come out no more knowledgeable, only more usable.

Q: How is DPO different from RLHF?

DPO, described in a 2023 paper by Rafailov and colleagues, uses the same kind of human preference data as RLHF but skips the separate reward model and reinforcement learning loop, training the language model directly on the preference pairs through a classification-style loss. It aims for the same outcome as RLHF with a simpler, more computationally efficient training process.

Q: What is RLAIF and how does it reduce the need for human labelers?

RLAIF, reinforcement learning from AI feedback, replaces much of the human ranking step in RLHF with feedback generated by another AI model applying a written set of principles, an approach introduced in Anthropic's 2022 Constitutional AI paper. Humans still write the guiding principles and evaluate final outputs, but far fewer individual response comparisons require direct human labeling.