From AlphaGo to StarCraft: How Reinforcement Learning Mastered Strategy Games

On March 9, 2016, something shifted in the relationship between human intelligence and machine intelligence. DeepMind's AlphaGo defeated 18-time world champion Lee Sedol at the ancient game of Go — a game long considered impossible for computers to master because its branching possibilities exceed the number of atoms in the observable universe. Over the next five games in Seoul, South Korea, AlphaGo won 4-1, and the shockwaves reached far past the Go community (Wikipedia: AlphaGo versus Lee Sedol). The moment wasn't just a milestone for AI. It was a revelation for games: the same training technique, reinforcement learning, could be pointed at any game with clear rules and a scoreboard, and the results kept getting stranger and more capable. In the decade since that match, reinforcement learning has gone on to conquer Chess, Shogi, Dota 2, StarCraft II, and even Diplomacy — a game built entirely on human negotiation — and the lessons from each conquest are now reshaping how professional players train and how studios design and balance their games.

Key takeaways

• Reinforcement learning (RL) trains an agent through trial-and-error reward signals rather than human-labeled examples, so it can discover strategies no human ever taught it.
• AlphaGo, AlphaZero, MuZero, OpenAI Five, AlphaStar, and CICERO each solved a progressively harder category of game: perfect information, then zero human data, then unknown rules, then hidden information and teamwork, then natural-language negotiation.
• These systems discovered novel strategies — new Go joseki, new Dota 2 team-fight timing, new StarCraft build orders — that professional players later adopted into competitive play.
• The same self-play and reward-modeling techniques are now used inside game studios for balance testing, exploit-hunting, and QA, not just for headline-grabbing matches.
• RL still struggles with very long time horizons, huge player counts, and games where "winning" is hard to define numerically, which is why 4X and grand-strategy titles remain largely unsolved.

What Is Reinforcement Learning, and Why Do Games Love It?

Reinforcement learning is a branch of machine learning where an agent learns by acting inside an environment rather than by studying labeled examples. There is no teacher marking each move "correct" or "incorrect." Instead, the agent takes an action, the environment returns a reward or penalty, and the agent adjusts its internal parameters to make rewarding actions more likely next time. Repeat that loop millions of times and a policy — the agent's rulebook for what to do in any given situation — starts to emerge on its own.

Think of it like training a dog with treats rather than handing it a manual. You never explain the rules of "sit" in words. You reward the behavior you want until the association forms. RL agents work the same way, except the "treats" are numerical reward signals (win the game: +1, lose the game: -1) and the "dog" is a neural network with millions or billions of tunable parameters.

Games are close to a perfect laboratory for this kind of learning because they have clear rules, a defined win condition, and an environment that can be simulated at scale, cheaply, in parallel, forever. An RL agent playing Go against a copy of itself doesn't need real human opponents, doesn't get tired, and doesn't need sleep. It can play more games in an afternoon on a data center than a human grandmaster plays in several lifetimes.

The earliest game-playing systems, by contrast, were narrow and brittle. IBM's Deep Blue beat world chess champion Garry Kasparov in 1997, but it did so through brute-force search of roughly 200 million positions per second, guided by an evaluation function that IBM's engineers had hand-tuned using decades of chess theory. Deep Blue didn't learn chess. It was told chess, one heuristic at a time. TD-Gammon, a backgammon-playing program built at IBM in the early 1990s by Gerald Tesauro, was the earlier and quieter ancestor of the modern approach — it used a form of temporal-difference learning to play itself and reached a level competitive with the world's top backgammon players, foreshadowing the self-play idea that would define AlphaGo two decades later.

Go has roughly 10^170 possible board positions — more than the number of atoms in the observable universe, which is estimated at around 10^80. Chess, by comparison, has "only" about 10^120 possible game trees, a number still large enough that brute-force search alone cannot solve it, which is why Deep Blue needed hand-built evaluation heuristics rather than pure search.

AlphaGo and the Move That Changed Everything

Go's astronomical position count is why the consensus among AI researchers, well into the 2010s, was that computers were a decade or more away from beating a Go professional. Brute-force search, the technique that had worked for Chess, was computationally impossible at Go's scale. Human players relied instead on intuition refined over decades of study — a kind of pattern recognition that felt uniquely human.

DeepMind's AlphaGo broke that assumption by combining deep neural networks with a search technique called Monte Carlo Tree Search (MCTS). One network, the "policy network," looked at a board position and predicted which of the legal moves were worth exploring further, narrowing the search dramatically. A second network, the "value network," estimated the probability of eventually winning from any given position, without having to play the game out to the end. Together, these two networks let AlphaGo search a far smaller slice of the game tree than brute force would require, while still finding strong, often surprising moves.

What stunned observers wasn't just that AlphaGo won. It was how it won. In game two against Lee Sedol, AlphaGo played what's now called Move 37: a shoulder hit on the fifth line, a position so unconventional that professional commentators assumed it was a mistake as it appeared on the board. DeepMind later said its own system had calculated only a 1-in-10,000 probability that a human professional would ever play that move. Lee Sedol reportedly left the room for roughly 15 minutes before returning to respond (Nerova: AlphaGo Move 37 Explained). Eighteen moves later, Go commentators realized the move wasn't a mistake at all — it had quietly reshaped the entire direction of the game in AlphaGo's favor. AlphaGo had found a line of play that centuries of accumulated human study had never explored.

AlphaZero, the generalized successor published by DeepMind in December 2017, took the idea a step further. It mastered Chess, Shogi (Japanese chess), and Go from scratch, using only the rules of each game plus self-play — no human game records, no opening books, no hand-tuned evaluation functions of any kind. Within 24 hours of self-play training, AlphaZero's Chess variant was already outperforming Stockfish, one of the strongest traditional chess engines in the world. Grandmasters who studied its games described its style as aggressive, materially generous, and positionally unlike anything in the existing chess literature, favoring long-term piece activity over the short-term material counting that had defined computer chess since Deep Blue.

DeepMind pushed the idea one step further still with MuZero, published in 2020, which learned to master Go, Chess, Shogi, and dozens of Atari games without even being told the rules in advance. Instead of being given the game's dynamics, MuZero learned an internal model of "what happens if I take this action" purely from experience, then planned inside that learned model. Where AlphaZero needed a perfect simulator of the game handed to it, MuZero built its own approximate simulator from scratch — a meaningfully harder and more general problem, closer to how a human infers the rules of an unfamiliar game just by playing it.

OpenAI Five: Cracking the Complexity of Team Play

Dota 2 is a fundamentally different kind of challenge than Go. Where Go is a perfect-information, two-player, turn-based game with a centuries-old body of theory, Dota 2 is a real-time, five-versus-five team game with hidden information (fog of war and unseen enemy inventories), 120-plus playable heroes each with unique ability kits, and a combined action and observation space that dwarfs Go's already enormous one. There is no clean way to enumerate "all possible Dota 2 games" the way researchers can approximate Go's position count.

OpenAI's answer was OpenAI Five, a system trained through self-play using Proximal Policy Optimization (PPO), a reinforcement learning algorithm well-suited to environments with continuous, high-dimensional action spaces. Training ran as a single continuous run from June 30, 2018 to April 22, 2019 — roughly ten months of real time — consuming about 770 ± 50 petaflop/s-days of compute and accumulating the equivalent of around 45,000 years of in-game Dota 2 experience, spread across 256 GPUs and 128,000 CPU cores (OpenAI: Dota 2 with Large Scale Deep Reinforcement Learning). The resulting model had more than 150 million parameters and learned to coordinate five separate agents as a single, role-aware team across 45-minute matches, without any human demonstrations to imitate.

On April 13, 2019, OpenAI Five defeated OG, the reigning back-to-back world champions at the time, 2-0 in a best-of-three exhibition series in San Francisco — the first time an AI system had beaten reigning world champions in a major esports title (OpenAI: OpenAI Five defeats Dota 2 world champions). What made the result notable to the Dota 2 community wasn't only the scoreline. In the days that followed, OpenAI opened the bots to the public in the "OpenAI Five Arena," where the system won 99.4 percent of roughly 7,257 games against human opponents (7,215 wins to 42 losses). Post-match analysis from professional Dota 2 commentators highlighted team-fight timing and objective-prioritization patterns that felt genuinely novel, and several ideas — particularly around when to commit a team fight around a jungle objective versus a lane push — made their way into competitive discussion in the following seasons.

How is Dota 2's problem different from Go's problem for an RL agent?

Go has full information and a small, discrete action space, so the hard part is search depth. Dota 2 has hidden information, five agents that must coordinate, and a continuous, real-time action space, so the hard part is credit assignment across teammates and time. OpenAI Five solved this with large-scale self-play and PPO rather than tree search, because MCTS-style search is far less tractable when the game state is only partially observable.

AlphaStar: Imperfect Information at Superhuman Speed

StarCraft II layers on challenges that neither Go nor Dota 2 fully capture at once. The map is partially hidden behind fog of war. The game runs continuously in real time rather than in discrete turns. And the action space is enormous: every unit, every building, every ability, across maps with thousands of possible camera positions and unit-selection combinations, all executed at up to several hundred actions per minute by top human players.

DeepMind's AlphaStar tackled this with a transformer-based architecture to encode the game state, a deep reinforcement learning policy trained with a novel off-policy algorithm, and — critically — a "league" training regime. Rather than training a single agent against itself, DeepMind maintained an entire population of agents that coevolved against one another, including specialist "exploiter" agents whose sole job was to find and punish weaknesses in the main agents' strategies. This prevented any single brittle strategy from dominating training, the same failure mode that had plagued earlier self-play systems that could get stuck exploiting a narrow, non-generalizing counter-strategy against themselves.

Published in Nature in November 2019, AlphaStar reached Grandmaster level — the top 0.2 percent of ranked human players — for all three StarCraft II races (Protoss, Terran, and Zerg), while playing on the official Battle.net ladder under the same interface restrictions as human players: no superhuman click speed advantage baked into the system, no camera or map information humans didn't also have access to (Nature: Grandmaster level in StarCraft II using multi-agent reinforcement learning). It was the first AI system to reach the top league of a widely played esport without any game-specific restrictions loosened in its favor, and it represents one of the clearest demonstrations to date of RL operating successfully under real-time, partial-information conditions.

Why did AlphaStar need a "league" of agents instead of one self-play loop?

A single agent training purely against itself can collapse into a narrow strategy that beats its own past selves but loses badly to a human playing something the agent never saw. DeepMind's league added dedicated "exploiter" agents whose only job was to find and punish these blind spots, forcing the main agents to develop broadly robust strategies rather than a single fragile counter-loop.

CICERO: When Reinforcement Learning Learned to Talk

Diplomacy is a strategy game almost entirely unlike the four above it. Seven players negotiate alliances, make promises, and then decide simultaneously whether to honor or betray them, with no dice, no hidden stats, and no combat resolution beyond raw troop-movement arithmetic. Winning requires persuading other humans to cooperate with you, in natural language, often while planning to betray them later. For years, researchers considered it a much harder target for AI than Go, Dota 2, or StarCraft II, because success depends on modeling what other people believe and want, not just on searching a game tree.

Meta's CICERO, published in Science in November 2022, combined a controllable dialogue model (a large language model fine-tuned to generate Diplomacy-relevant negotiation text) with a separate strategic reasoning engine that used planning and reinforcement learning to decide what CICERO should actually propose and whether to honor its own promises. The dialogue model handled the "talking" and the planning engine handled the "thinking," with each one constraining the other so CICERO's messages stayed consistent with a coherent, achievable plan rather than drifting into promises it had no intention of keeping. Across 40 games in an anonymous online Diplomacy league, CICERO achieved more than double the average score of human players and placed first in an eight-game tournament of 21 participants, ranking in the top 10 percent of players who played more than one game (Science: Human-Level Play in the Game of Diplomacy). CICERO is worth including alongside AlphaGo and AlphaStar precisely because it shows RL techniques generalizing past pure mechanical execution and into a domain built on trust, persuasion, and language.

CICERO achieved more than double the average score of human players across 40 games in an anonymous online Diplomacy league and finished first in an eight-player tournament — all while a majority of its opponents never suspected they were negotiating with a machine, according to Meta's Science publication.

Comparing the Landmark Systems

Each of these systems tackled a different combination of constraints. Laid side by side, the progression from Go to Diplomacy is really a progression through increasingly realistic, increasingly human forms of difficulty: from perfect information to imperfect information, from a single opponent to team coordination, from a fixed rulebook to no rulebook at all, and finally to natural language itself.

Table: Landmark reinforcement learning game-playing systems, 2016–2022

System	Game	Developer	Year	Core technique	Headline result
AlphaGo	Go	DeepMind	2016	Policy/value networks + MCTS	Beat Lee Sedol 4-1
AlphaZero	Chess, Shogi, Go	DeepMind	2017	Self-play, zero human data	Outperformed Stockfish within 24 hours of training
MuZero	Go, Chess, Shogi, Atari	DeepMind	2020	Learned world model + planning	Mastered games without being given the rules
OpenAI Five	Dota 2	OpenAI	2019	Large-scale self-play with PPO	Beat world champions OG 2-0
AlphaStar	StarCraft II	DeepMind	2019	Multi-agent league training	Reached Grandmaster level, all three races
CICERO	Diplomacy	Meta AI (FAIR)	2022	Language model + strategic planning	More than double average human score across 40 games

Sources: DeepMind, OpenAI, and Meta AI publications cited throughout this article.

From Research Demo to Development Pipeline

The most durable legacy of these systems isn't the highlight-reel matches. It's that the underlying techniques — self-play, reward modeling, large-scale simulation — have quietly become practical tools inside game studios, not just headline-grabbing research demos.

An RL agent trained on a strategy game is, in effect, an automated strategy tester that never gets bored and never plays favorites. It explores a game's strategy space far more exhaustively than any human playtesting team could, and it does so without the biases a human tester brings from their own playstyle preferences. Where a QA team might spend weeks probing a new patch for game-breaking exploits, an RL agent can run millions of simulated matches overnight and flag the specific interaction that breaks the balance.

This shift is already visible in how major studios operate. Blizzard has used automated self-play systems to stress-test StarCraft II balance patches before public release, catching dominant strategies before players find them in the wild. Valve has used agent-based analysis on Dota 2 to identify mechanics that could be exploited at a level beyond normal human play patterns. The broader pattern across the industry is that RL agents are increasingly treated the way a compiler treats code: an automated check that runs before a human ever sees the result, catching the kind of systemic imbalance that's nearly invisible from inside a single human playthrough but glaringly obvious once an agent has played the same matchup a million times.

The techniques are also reshaping single-player and NPC design conversations, even outside strict RL research. A system that can learn to play at a target skill level, rather than following a scripted difficulty curve, opens the door to adaptive opponents that scale to a player's actual ability in real time instead of a fixed "easy/medium/hard" toggle. That idea connects directly to the broader machine learning landscape covered in SciHub101's complete machine learning guide, where reinforcement learning sits alongside supervised and unsupervised learning as one of the three foundational paradigms.

What People Get Wrong About Game-Playing AI

The biggest misconception is that these systems are "just very good at search," a holdover from the Deep Blue era. Deep Blue really was mostly search: fast, exhaustive, guided by rules humans wrote. AlphaGo, AlphaStar, and CICERO are fundamentally different — the neural networks inside them learned their own evaluation of a position, their own sense of "this looks promising," from experience rather than from a programmer's checklist. That distinction is why AlphaGo's Move 37 surprised its own creators as much as it surprised Lee Sedol: nobody at DeepMind wrote a rule that produced that move. The network learned it.

A second common misconception is that RL agents are "creative" in the way a human artist or strategist is creative. They aren't generating novelty for its own sake. They're the product of an optimization process relentlessly hunting for whatever maximizes a numerical reward signal, and sometimes that hunt lands on a strategy no human considered simply because humans, guided by intuition and social convention, never bothered to search that part of the space. It looks like creativity from the outside. Mechanically, it's closer to an extremely patient, extremely thorough form of trial and error — more analogous to evolution finding an unexpected adaptation than to a novelist choosing a striking metaphor.

A third misconception, especially common among players encountering RL-trained opponents for the first time, is that "superhuman" means "unbeatable" or "plays like a human, only better." AlphaStar was restricted to human-legal actions per minute and still occasionally made decisions that a strong human would consider strange, because its notion of "strong" was shaped entirely by what maximized win probability in its training population, not by aesthetic or conventional judgment. This is one reason RL opponents tuned purely to maximize win rate can feel unsatisfying to play against in a commercial game: winning and being fun to lose to are different objectives, and reward functions have to be designed with both in mind if the goal is a good player experience rather than a research benchmark.

Where Reinforcement Learning Still Struggles

For all the landmark wins, RL in games has clear limits, and it's worth being direct about them rather than treating every strategy game as a matter of time and compute.

Long time horizons remain difficult. A 45-minute Dota 2 match or a StarCraft ladder game is long by RL standards, but a Civilization-style 4X game can run for hundreds of turns across an entire tech tree and centuries of in-game history, with consequences from an early decision surfacing fifty turns later. Assigning credit for a win or loss back to the specific decision that caused it — the "credit assignment problem" — gets exponentially harder as the time horizon stretches.

Player count is another wall. Every system covered here tops out at seven players (CICERO's Diplomacy), and even that required a specialized architecture combining language modeling with planning. Games with eight or more simultaneous human-scale agents, especially ones involving shifting coalitions and social dynamics, remain largely unsolved, because the space of possible alliances and betrayals grows combinatorially with each additional player.

Reward design itself is a persistent bottleneck. Go, Chess, and StarCraft's ladder all have an unambiguous win condition that's easy to turn into a reward signal: 1 for a win, -1 for a loss. Games without a clean numeric win condition — open-ended sandbox titles, cooperative narrative games, anything where "success" is subjective or social — don't have an obvious reward function to optimize, and poorly designed reward functions are notorious for producing agents that technically maximize the number they were given while behaving in ways nobody actually wanted, a failure mode researchers call reward hacking.

The Takeaway

The through-line from AlphaGo to CICERO isn't that machines got better at playing games. It's that the definition of what counts as a "solvable" game kept expanding, from Go's perfect information, to Dota 2's hidden information and teamwork, to StarCraft's real-time execution, to Diplomacy's reliance on trust and language itself. Each system inherited the same core idea — learn from reward, not from labeled examples — and each had to be redesigned around what made its particular game hard. The practical legacy for players and developers alike is less about superhuman highlight matches and more about the quieter shift already underway inside studios: self-play agents finding exploits before players do, adaptive difficulty that responds to skill rather than a fixed slider, and a research pipeline that keeps discovering the strategy space games have always contained but human play alone was never going to fully explore.

FAQ

Q: How does AlphaGo's reinforcement learning differ from how humans learn strategy games? A: Humans learn from studying example games, receiving coaching, and playing a limited number of games over years. AlphaGo played millions of games against itself in simulation, learning purely from the outcome signal (win or loss) without any human guidance on what good play looks like. AlphaZero went further and removed human game data entirely, learning Chess, Shogi, and Go from the rules alone.

Q: Did AlphaGo and AlphaStar actually change how professionals play their games? A: Yes. AlphaGo's unconventional joseki (corner sequences) in Go, including patterns from Move 37, were studied and partially adopted by professional players worldwide. AlphaStar's build orders and unit compositions influenced competitive StarCraft II discussion. OpenAI Five's team-fight timing and objective-prioritization patterns influenced Dota 2 strategy conversations following its 2019 win over OG.

Q: Could an RL agent master any strategy game given enough compute? A: In theory, given a well-defined reward signal and enough training time, many games are approachable. In practice, games with very long time horizons, large player counts, and ambiguous win conditions remain far harder to reward-model correctly, which is why 4X games like Civilization and grand strategy titles like Europa Universalis have no AlphaGo-equivalent yet.

Q: Can reinforcement learning be used to create fun AI opponents in commercial games? A: Potentially, yes, but it requires deliberately designing for "fun to play against," not just "wins." RL agents optimized purely to maximize win rate can develop strategies that feel unfair, repetitive, or unnatural to human opponents. Balancing raw competence with an enjoyable opponent experience is an active area of game AI research rather than a solved problem.

Q: What is the difference between AlphaZero and MuZero? A: AlphaZero is given the complete rules of the game in advance — it knows exactly what happens if it plays a given move — and uses self-play to learn a strong policy within those known rules. MuZero is not given the rules at all. It learns its own internal model of how the game behaves purely from experience, then plans using that learned model, which is a meaningfully harder and more general problem.

Q: How much computing power did OpenAI Five actually use to beat OG? A: OpenAI Five trained for roughly ten months, from June 30, 2018 to April 22, 2019, consuming about 770 ± 50 petaflop/s-days of compute across 256 GPUs and 128,000 CPU cores, and accumulating the equivalent of around 45,000 years of in-game Dota 2 experience.

Q: Is CICERO actually reading human emotions or intentions? A: No. CICERO models what other players are likely to do next based on the conversation and game state, using a planning engine, and generates dialogue consistent with a coherent strategy. It does not have access to emotional or biometric signals — its "understanding" of intent comes entirely from text and observed in-game moves, published in its 2022 Science paper.

Q: Do game studios actually use these techniques, or is it just research? A: Both. AlphaGo, AlphaStar, OpenAI Five, and CICERO were research projects, but the underlying self-play and reward-modeling techniques have moved into production pipelines. Blizzard has used automated self-play to stress-test StarCraft II balance patches, and Valve has used agent-based analysis to identify exploitable mechanics in Dota 2 ahead of patches.

Q: What strategy game challenges remain unsolved for AI? A: Large-scale 4X games (Civilization-style), grand strategy games (Europa Universalis), and real-time games with eight or more simultaneous players remain difficult, largely because of long time horizons and combinatorial alliance dynamics. Diplomacy has been approached with human-level competitiveness by CICERO, but full mastery of negotiation and deception at the highest human level, across more varied social contexts, is still an open research area.

Q: Why did it take until 2019 for an AI to reach Grandmaster level in StarCraft II when AlphaGo beat a Go champion in 2016? A: StarCraft II combines several hard problems Go doesn't have: hidden information behind fog of war, real-time execution instead of turns, and a vastly larger combined action and observation space. DeepMind had to develop an entirely new training approach, multi-agent league training with dedicated exploiter agents, rather than reusing AlphaGo's Monte Carlo Tree Search approach, which is far less effective under partial information.

---

Read next:

• Complete AI Guide 2026
• Complete Machine Learning Guide 2026
• What Is MLX? Apple's ML Framework