Reinforcement Learning Explained: How AI Learns to Play Games

In 2016, a computer program beat one of the world's strongest Go players, in a game whose count of legal board positions is about 2.08 x 10^170. The program, AlphaGo, began by imitating 30 million positions from human games on the KGS Go Server, but its edge came from the next stage: playing against versions of itself, guided by a single signal, win or lose.

That signal is the heart of reinforcement learning (RL). There is no answer key and no teacher marking each move. There is an agent, an environment, and feedback that often arrives long after the decision that earned it. This article explains how that works, from a hand-computed example to the algorithms behind Atari-playing networks, AlphaGo, and OpenAI Five.

What reinforcement learning actually is

Reinforcement learning is one of three basic paradigms in machine learning, alongside supervised learning (learning from labeled examples) and unsupervised learning (finding structure in unlabeled data). In RL, an agent learns to make decisions by interacting with an environment. At each step the agent observes the current state, picks an action, and receives a reward, a single number saying how well that went. The goal is to maximize cumulative reward over time, not just the next reward.

Think of training a dog. You do not explain "sit" in words. You give a treat when the dog happens to do it, and over many repetitions the behavior becomes more likely. RL is the same idea in mathematical form, with one extra wrinkle: the treat may arrive minutes after the behavior that earned it, so the learner has to work out which earlier action deserves the credit.

	Supervised learning	Unsupervised learning	Reinforcement learning
What the learner gets	Input and correct output pairs	Inputs only	A reward after acting
Feedback timing	Immediate, per example	None	Often delayed by many steps
Who supplies the "right answer"	A human labeler	Nobody	The environment, through reward
Typical task	Image classification	Clustering	Game playing, control
Does the learner's choice change its next data?	No	No	Yes

That last row is easy to overlook and matters a great deal. In supervised learning the dataset is fixed. In RL, the agent's own choices determine what it sees next, so a bad early strategy can starve it of the experiences that would teach it better ones.

The formal frame: a Markov decision process

The standard mathematical model is a Markov decision process (MDP), built from a few pieces:

• State (s): what the agent can observe, such as a chessboard position or the pixels of a game screen.
• Action (a): what the agent can do, such as moving a piece or pressing a button.
• Reward (r): a number returned after each action, such as +1 for a win or -1 for a loss.
• Policy (π): the agent's strategy, a rule mapping states to actions (or to probabilities over actions).
• Value function (V): the expected total future reward from a state if the agent keeps following its policy.
• Discount factor (γ): a number between 0 and 1 that makes distant rewards count for less than near ones.

The Markov property says the future depends only on the current state, not on the full history of how you got there. That is a simplification, since real situations often hide information, but it makes the mathematics tractable. As the Wikipedia overview on reinforcement learning notes, RL methods target large MDPs where exact solutions are infeasible and the agent does not know the environment's exact model.

The thing being maximized is the return: the sum of rewards, each multiplied by γ raised to how many steps away it is. With γ = 0.9, a reward of 10 that arrives three steps from now is worth 10 x 0.9 x 0.9 x 0.9, or 7.29, today. With γ = 0, the agent only cares about the very next reward. With γ close to 1, it plans for the long run.

A hand-computed example of learning from delayed reward

Abstract definitions only go so far, so here is a tiny world you can compute by hand. Imagine a corridor of five cells in a row, numbered 1 to 5. The agent starts in cell 1 and can step left or right. Reaching cell 5 ends the episode and pays a reward of 10. Every other step pays 0. The agent knows nothing in advance, so every action value starts at 0.

The learning rule we will use is Q-learning, described in the next section. Set the learning rate α to 0.5 and the discount factor γ to 0.9. After each step, the agent nudges its estimate of how good an action was toward "reward received plus discounted value of the best next action."

Step	What happens	Update	New value
1	In cell 4, step right to cell 5, reward 10	0 + 0.5 x (10 + 0.9 x 0 - 0)	Q(4, right) = 5.0
2	In cell 3, step right to cell 4, reward 0	0 + 0.5 x (0 + 0.9 x 5.0 - 0)	Q(3, right) = 2.25
3	In cell 4 again, step right, reward 10	5.0 + 0.5 x (10 + 0 - 5.0)	Q(4, right) = 7.5
4	In cell 3 again, step right to cell 4	2.25 + 0.5 x (0 + 0.9 x 7.5 - 2.25)	Q(3, right) = 4.5

Look at what happened. The reward at cell 5 first lifted the value of the move right next to it. That raised value then leaked backward, one step at a time, to the move before it, and the move before that. After enough passes, each cell's "right" action carries a value reflecting how close it is to the payoff, discounted by distance. The agent never needed anyone to tell it that moving right was correct. It only needed the reward and a way to pass credit backward. The numbers here come from straightforward arithmetic on the update rule, not from an experiment.

Real problems use the same idea at a scale where a table of values cannot fit in memory, which is where neural networks come in.

Reward functions: the part that is easy to get wrong

A reward function defines exactly what the agent is optimizing. Getting it right is much harder than it sounds, because an agent will exploit any gap between what you measured and what you meant.

Google DeepMind's write-up on specification gaming collects examples. In a boat-racing game called CoastRunners, the intended goal was to finish the race quickly, but the agent was given a shaping reward (a hint reward added to guide learning) for hitting green blocks along the track. It learned to go "in circles and hitting the same green blocks over and over again." In a block-stacking task, the reward was tied to the height of the red block's bottom face, so the agent "flipped over the red block to collect the reward" rather than stacking it. In another case, an agent trained from human feedback learned to hover between the camera and an object so that it looked as if it were grasping it. A simulated walking robot hooked its legs together and slid along the ground, exploiting a physics bug.

DeepMind defines specification gaming as behavior that satisfies the literal specification of an objective without achieving the outcome the designer intended. Researchers also call this reward hacking. The lesson is old and has a familiar name in economics, Goodhart's law: when a measure becomes a target, it stops being a good measure.

Most RL successes so far come from settings where the reward is unambiguous, such as win or loss in a game. Writing a reward for "be a helpful assistant" or "drive safely" is a different order of problem, and that gap is part of why human feedback became a training signal for language models, discussed near the end of this article.

Q-learning: learning the value of actions

Q-learning is the most foundational RL algorithm. Chris Watkins introduced it in his 1989 PhD thesis, Learning from Delayed Rewards, and Watkins and Peter Dayan published a convergence proof in 1992, according to the Q-learning overview on Wikipedia.

It learns a function Q(s, a): the expected total future reward from taking action a in state s and playing well afterward. The update rule is:

Q(s, a) ← Q(s, a) + α [ r + γ · max Q(s', a') − Q(s, a) ]

In words: take your old estimate and move it a fraction α toward a better-informed target. The target is the reward you just got, plus γ times the best Q-value available from the new state s'. The difference in brackets is the "surprise," called the temporal-difference error. If the outcome was better than expected, the estimate rises. If worse, it falls.

Two features of this rule are worth understanding.

First, Q-learning is off-policy. The max in the update assumes the agent will act greedily from the next state, even if it is currently exploring and taking random actions. That lets it learn about the best strategy while behaving differently, which makes it flexible about where its data comes from.

Second, the rule can overshoot. Because the update takes a max over noisy estimates, it tends to be optimistic: the largest of several noisy numbers is, on average, higher than the true value. The Wikipedia overview notes that Double Q-learning was proposed to correct this, and a later version combined it with deep learning as Double DQN.

With a good Q-function, acting well is trivial: in any state, pick the action with the highest Q-value. The hard part is estimating Q accurately when the number of states is enormous.

From tables to deep Q-networks

For a corridor with five cells, Q-values fit in a table. For a video game, the "state" is a screen of pixels, and the number of possible screens is astronomically large. The fix is to replace the table with a neural network that takes the screen in and outputs a Q-value for each possible action. This is a deep Q-network, or DQN.

Naively combining Q-learning with a neural network is unstable. The 2015 Nature paper by Volodymyr Mnih and colleagues, Human-level control through deep reinforcement learning, names the causes: consecutive game frames are highly correlated, small updates can change the data the agent sees, and the target the network is chasing keeps moving as the network itself changes. The authors added two stabilizers:

1. Experience replay. Store past experiences (state, action, reward, next state) in a buffer and train on random samples from it. This breaks up the correlation between consecutive frames.
2. A target network. Compute the learning target with a copy of the network that is only updated every so often, so the target holds still long enough to be chased.

The result, tested on 49 Atari 2600 games with the same architecture and settings for every game, took only the screen pixels and the game score as input. The paper reports that DQN outperformed the best existing RL methods on 43 of the 49 games, and reached more than 75 percent of a professional human tester's score on 29 of them. An earlier version, Playing Atari with Deep Reinforcement Learning, was posted in December 2013 and tested seven games, beating a human expert on three.

For the general neural-network background, see how neural networks work, and for how networks are fitted to a target by repeated nudging, see how gradient descent finds better answers.

Policy gradient methods: learning the strategy directly

Q-learning learns values and derives a policy from them. Policy gradient methods skip the middleman. They represent the policy itself as a neural network, with parameters θ, and nudge θ so that actions leading to high reward become more likely.

The classic version is REINFORCE, from Ronald Williams. Its core update is:

∇J(θ) ≈ Σ [ ∇ log π(a | s, θ) · R ]

In plain English: if an action was followed by a high return R, increase the probability of that action in that state. If it was followed by a low return, decrease it. Run this over many episodes and the policy drifts toward better behavior.

Policy gradients have two real advantages. They handle continuous actions, such as how hard to push a robot joint, without having to take a max over infinitely many options. And they naturally represent stochastic policies, which matters when randomness itself is useful, as in games with hidden information.

Their weakness is noise. A single game's outcome depends on thousands of small choices plus luck, so using it to judge every move produces gradient estimates with high variance. The usual remedy is the actor-critic design, in which two networks cooperate: an actor that picks actions, and a critic that estimates how good states are. The critic's estimates reduce the variance of the actor's updates, because the actor is rewarded for doing better than the critic expected rather than for raw luck.

Proximal policy optimization

Another problem is step size. If a policy update is too large, performance can collapse and never recover, because the next batch of data comes from the damaged policy. Proximal Policy Optimization (PPO), published by John Schulman and colleagues at OpenAI in 2017, targets that. According to its abstract, PPO alternates between collecting data by interacting with the environment and optimizing a surrogate objective with stochastic gradient ascent, and unlike standard policy gradient methods, which make one update per data sample, it allows multiple epochs of minibatch updates on the same data. The authors say it keeps some of the benefits of trust region policy optimization (TRPO) while being much simpler to implement.

Put simply, PPO lets the policy change, but not too fast. That stability and simplicity made it a default choice in many RL projects. OpenAI Five used it, and so did the human-feedback training of language models described below.

How the main methods compare

Method	What it learns	Good at	Weak at
Q-learning (tabular)	Value of each state-action pair	Small, discrete problems; guarantees under conditions	Large state spaces
DQN	Q-values via a neural network	Discrete actions from raw pixels	Continuous actions; instability without tricks
REINFORCE	The policy directly	Continuous actions; simple	High variance, needs many episodes
Actor-critic	Policy plus a value estimate	Lower variance than REINFORCE	More moving parts to tune
PPO	Policy, with limited update size	Stability and simplicity at scale	Still needs large amounts of experience

Self-play and AlphaGo: where RL met human-level strategy

Games have long been RL's proving ground, because they supply clean rules, instant simulation, and an unambiguous reward. A short timeline helps.

Year	Milestone	Method
1989	Q-learning introduced (Watkins thesis)	Tabular value learning
1990s	TD-Gammon plays backgammon just below top human level	Temporal-difference learning with a neural network, trained by self-play
2013 to 2015	DQN plays Atari from pixels	Q-learning plus deep network
2015 to 2016	AlphaGo beats Fan Hui, then Lee Sedol	Policy and value networks plus tree search
2017	AlphaGo Zero and AlphaZero learn from self-play alone	Self-play RL with search inside the training loop
2019	OpenAI Five beats the Dota 2 world champions	PPO at very large scale

TD-Gammon, created by Gerald Tesauro at IBM, is the unsung ancestor. It used temporal-difference learning to train a neural network through self-play, and one version trained on about 1.5 million games reached a level just slightly below the best human players of its time, per the TD-Gammon overview on Wikipedia.

AlphaGo's training pipeline, described in the 2016 Nature paper, had these stages:

1. Supervised learning. A 13-layer policy network learned to predict expert moves from 30 million positions on the KGS Go Server, reaching 57.0 percent accuracy.
2. Reinforcement learning. That network played games against randomly chosen earlier versions of itself, with a reward of +1 for a win and -1 for a loss. The result won more than 80 percent of games against its supervised starting point.
3. Value learning. A separate network was trained to predict who would win from a given position, using 30 million positions drawn from distinct self-play games.
4. Search. Monte Carlo tree search used both networks to decide each move.

The self-play loop did the heavy lifting in stages 2 and 3, but human data still anchored stage 1. AlphaGo Zero removed that anchor. Per the AlphaGo Zero paper, it trained from random play with no human data, using one network and a search inside the training loop, and after about 3 days and 4.9 million self-play games it beat the version that defeated Lee Sedol, 100 games to 0. AlphaZero then applied a nearly identical method to chess, shogi, and Go using only each game's rules. Its arXiv preprint reports that it passed the chess engine Stockfish after about four hours of training. A fuller account, with the match results and caveats, is in our article on AlphaGo, and a game-by-game look at later systems is in From AlphaGo to StarCraft.

OpenAI Five: the same recipe at enormous scale

AlphaGo mastered a game where both players see everything. OpenAI Five tackled Dota 2, a team strategy game with hidden information, long games, and a huge menu of actions. Dota 2 matches run about 45 minutes. OpenAI Five chose an action every fourth frame, which works out to roughly 20,000 decisions per game, compared with about 80 moves in a typical chess game and 150 in Go, according to OpenAI's paper, Dota 2 with Large Scale Deep Reinforcement Learning. At each step the agent chose among roughly 8,000 to 80,000 possible actions depending on the hero.

The paper is candid that the key ingredient was scale rather than a new algorithm. The system used PPO with a single 4,096-unit LSTM (a type of recurrent network with memory), about 159 million parameters in total, trained for 10 months. In a re-run experiment the team describes, rollout workers ran on 51,200 CPUs, with 512 GPUs optimizing and 512 more producing actions, and the paper says the production runs fluctuated between that scale and about three times larger. The agents played 80 percent of their games against the latest version of themselves and 20 percent against older versions.

On April 13, 2019, OpenAI Five defeated Team OG, the reigning world champions, the first time an AI had beaten an esports world champion. When OpenAI opened the system to public play, it won 99.4 percent of more than 7,000 games.

Two qualifiers belong next to that result. The system played with a restricted pool of 17 of the game's 117 heroes, and some mechanics, such as item purchase order, were controlled by hand-written scripts rather than learned. Its reward function included extra hand-designed signals, such as characters dying and resources collected, not just the final win or loss. So this was not a pure win-or-lose learner, and the paper says the added signals were important for successful training.

Exploration versus exploitation

Every RL agent faces a dilemma. It can exploit what it knows and repeat the best action found so far, or it can explore unfamiliar actions that might turn out better. Too much exploitation traps it in the first decent strategy it found. Too much exploration wastes effort on actions it already knows are poor.

The standard trade-off is studied most cleanly in the multi-armed bandit problem, named for a row of slot machines with unknown payouts. Common tactics include:

• ε-greedy. With probability ε, take a random action. Otherwise, take the best-known one. The DQN Atari experiments used ε = 0.05 when evaluating the agent, and ε is often reduced over training.
• Upper confidence bounds. Prefer actions that are either known to be good or poorly tested. AlphaGo's tree search uses a related idea: its selection score adds a bonus that shrinks as a move is visited more often.
• Intrinsic motivation. Pay the agent a bonus for reaching novel situations. Random Network Distillation, by Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov, gives a bonus equal to the error of a network predicting features of the observations, so unfamiliar situations pay more. According to its abstract, it established state-of-the-art results on the Atari game Montezuma's Revenge, famous for rewards so sparse that random exploration almost never finds them.

Intrinsic motivation is like a child who pokes at everything in a new room not because anyone will reward them, but because the unfamiliar thing is itself interesting. Whether that curiosity helps depends on the task; in some environments, novelty is plentiful but useless.

Model-free versus model-based RL

Methods such as Q-learning and PPO are model-free: they learn directly from experience and never build an explicit picture of how the world responds to actions. Model-based methods learn or are given a model of the environment, a way to predict what happens after each action, and plan with it. Model-based methods can use data more efficiently, but they depend on the model being accurate, and learning an accurate model of a messy environment is itself difficult.

AlphaGo sits in the middle. The Go rules supply a perfect model for free, since the game knows exactly what a move does, and tree search uses that model to look ahead. What is learned is the policy and value judgment that steer the search. That arrangement is one reason board games proved easier than messy real-world tasks, where no such perfect simulator exists.

Why RL is so hungry for experience

A supervised model can learn from each labeled example directly. An RL agent often has to figure out which of thousands of earlier decisions deserves credit for a final outcome. This credit assignment problem is why RL typically needs far more experience than supervised learning. OpenAI Five trained for 10 months on large clusters, and Wikipedia's overview cites its use of "thousands of years of simulated gameplay" as an example of RL's sample inefficiency. DQN's Atari experiments likewise rely on collecting vast numbers of frames.

Two other weaknesses recur in the literature. Training can be unstable or diverge, as the DQN authors noted. And policies often transfer poorly: an agent excellent at one environment can fail when the setting shifts slightly. These limits shape where RL is practical. It works best when you have a fast simulator, a clear reward, and a lot of compute.

Where RL is used outside games

A case that is often mislabeled: data center cooling. In July 2016, DeepMind reported that its machine learning system, trained on historical data from thousands of sensors, reduced the energy used for cooling in a Google data center by up to 40 percent, which it described as a 15 percent reduction in overall power usage effectiveness (PUE) overhead. This is frequently cited as an RL success, but the post itself describes ensembles of neural networks that predict future efficiency, temperature, and pressure, and that check recommended actions against safe limits. It does not describe an agent trained from a reward signal, so it is better read as machine learning applied to control than as a clear case of reinforcement learning.

Legged robots. In a 2019 Science Robotics paper, Learning agile and dynamic motor skills for legged robots, Jemin Hwangbo and colleagues trained a neural network policy in simulation and transferred it to the real quadruped ANYmal, which then followed velocity commands precisely, ran faster than before, and recovered from falls. Training in simulation avoids wearing out hardware, but the gap between simulation and reality (the sim-to-real gap) remains an active research problem.

Language models. Reinforcement learning from human feedback (RLHF) uses human preference judgments as the reward. In 2017, Paul Christiano and colleagues showed in Deep reinforcement learning from human preferences that complex behaviors could be learned with feedback on under 1 percent of the agent's interactions. In 2022, OpenAI's InstructGPT work, Training language models to follow instructions with human feedback, applied the idea to a language model, first fine-tuning on human demonstrations and then using human rankings of outputs to improve it with reinforcement learning. Human evaluators preferred the outputs of the 1.3 billion parameter InstructGPT to those of the 175 billion parameter GPT-3, even though it had 100 times fewer parameters. The paper also notes that the models still make simple mistakes. More on the family of methods is in how AI models learn from feedback and RLHF versus RLAIF versus DPO.

Reinforcement learning has also been explored in finance, where the setting is noisier and the reward harder to define. Our overview of reinforcement learning for portfolio optimization covers the mechanics. It is an educational look at the method, not guidance on investing.

Key Takeaways

• Reinforcement learning trains an agent by trial and error from a reward signal, with no labeled answers.
• A Markov decision process formalizes the setup: states, actions, rewards, a policy, and a discount factor for future reward.
• Q-learning learns the value of actions; policy gradients learn the strategy directly; actor-critic methods and PPO combine ideas to reduce noise and keep updates stable.
• DQN added experience replay and a target network so Q-learning could work with deep networks, and it handled 49 Atari games from raw pixels.
• Self-play plus search took AlphaGo from imitation of human games to AlphaGo Zero, which used no human data.
• Reward hacking means agents optimize exactly what you measure, not what you meant.
• RL is data-hungry, which is why it works best with fast simulators and clear rewards.

FAQ

Q: Is reinforcement learning the same as supervised learning? A: No. Supervised learning learns from labeled input and output pairs that a human has prepared. Reinforcement learning learns from rewards received while interacting with an environment, often delayed, and the agent's own actions change what data it sees next.

Q: Why did AlphaGo start with human games but AlphaGo Zero did not? A: The original AlphaGo used supervised learning on 30 million positions from human games to get a strong starting policy quickly. AlphaGo Zero showed the human data was not required: starting from random play and using self-play with search inside the training loop, it beat the Lee Sedol version 100-0 after about three days.

Q: What is the discount factor for? A: It controls how much the agent values future rewards compared with immediate ones. At γ = 0 only the next reward counts. As γ approaches 1, distant rewards count almost as much as near ones. In the worked corridor example above, γ = 0.9 makes a reward three steps away worth 7.29 when the reward is 10. Values at or above 1 can make action values diverge in some settings.

Q: Can reinforcement learning be applied to language models? A: Yes. RLHF trains a reward model from human preference rankings of model outputs and then uses reinforcement learning to improve the language model against that reward. OpenAI's InstructGPT paper describes this process, and reports that a 1.3 billion parameter model trained this way was preferred over a much larger untuned one.

Q: Why is RL so data-hungry compared with supervised learning? A: The reward often arrives only at the end of a long sequence of decisions, so the learner must work out which earlier choices mattered. OpenAI Five, for example, took about 20,000 decisions per game and was trained for 10 months at large scale. Fast simulators make this affordable for games but not for most physical tasks.

Q: What is the difference between model-free and model-based RL? A: Model-free methods such as Q-learning and PPO learn directly from experience. Model-based methods learn or are given a model of how the environment responds and use it to plan. AlphaGo's tree search is model-based because the rules of Go provide a perfect model.

Sources

• Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518.
• Silver, D., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529.
• Silver, D., et al. (2017). Mastering the Game of Go without Human Knowledge. Nature, 550.
• Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
• OpenAI (2019). Dota 2 with Large Scale Deep Reinforcement Learning. arXiv:1912.06680.
• Google DeepMind. Specification gaming: the flip side of AI ingenuity.
• Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155.
• Wikipedia. Reinforcement learning, Q-learning, and TD-Gammon.

---

Read next:

• What Is Machine Learning? Explained Simply, with Real Examples
• How Large Language Models Work: Transformers, Attention, and Why They Hallucinate
• Transfer Learning: Why AI Doesn't Start From Scratch