Reinforcement Learning for Robotics: Teaching Machines to Move Through the World

A four-legged robot can learn to walk on flat ground in under four minutes and on uneven terrain in twenty, using a single workstation GPU. The catch is that almost all of that learning happens in a simulator, where thousands of virtual copies of the robot fall over, get up, and try again. Reinforcement learning for robotics is built on that trade: millions of cheap simulated failures in place of a few expensive real ones, then a careful jump to hardware. The rest of the field is about whether that jump holds.

Key takeaways

• Reinforcement learning (RL) trains a policy, a function that maps sensor readings to motor commands, by rewarding good outcomes and penalizing bad ones.
• Real robots wear out and need human help to reset, so nearly all RL training for locomotion happens in physics simulation, then transfers to hardware.
• Domain randomization, which varies friction, mass, lighting, and other settings during training, is the main way to make a policy survive the gap between simulation and reality.
• Legged locomotion is the clearest success story. Manipulation relies more on imitation learning from large demonstration datasets, with RL playing a supporting role.
• Reward design, contact-rich tasks, and long-run reliability remain unsolved, and many headline demos blend learned and classical control.

How robot RL works, in plain terms

Start with the loop. A robot, called the agent, observes its situation: camera pixels, joint angles, the readings from an inertial sensor that tracks tilt. It picks an action, such as a torque for each motor. The world responds, and a reward function scores the result. The agent's job is to find a policy that collects as much reward as possible over time.

An analogy: learning to ride a bicycle. Nobody gives you the equations for balance. You wobble, correct, fall, and slowly your body learns which small steering adjustments keep you upright. RL does the same thing numerically, with a neural network standing in for muscle memory. The difference is that a person falls off a bike a few dozen times. A robot policy may need to fall millions of times, which is the central problem of the field.

Why real hardware is a poor classroom

Learning in the physical world is slow in ways that software never is. Motors heat up and gears wear, and somebody has to pick the robot up, put the objects back, and press start again. Collecting data at scale is a project in its own right. For a sense of the cost, the RT-1 robot model was trained on roughly 130,000 demonstration episodes, collected over 17 months with 13 robots (Brohan et al., RT-1, arXiv:2212.06817). That is demonstration data from people, not trial-and-error learning, and it already took more than a year.

A simulator removes the human from the loop. A policy can attempt a task at many times real-time speed, on as many virtual robots as the hardware can hold. OpenAI's Rubik's Cube hand policy, discussed below, used roughly 13 thousand years of cumulative simulated experience (Akkaya et al., 2019). No physical robot could accumulate that.

The simulators

Simulation software for robots models rigid bodies, joints, contacts, and friction. A few names come up repeatedly:

• MuJoCo. Developed in the Movement Control Laboratory at the University of Washington and described in a 2012 paper by Emanuel Todorov, Tom Erez, and Yuval Tassa. It was commercialized, acquired by Google DeepMind in October 2021, and open-sourced under the Apache 2.0 license in May 2022 (Wikipedia, MuJoCo).
• Isaac Gym and Isaac Lab (NVIDIA). The 2021 Isaac Gym paper reported 2 to 3 orders of magnitude improvements over conventional RL setups that simulate on the CPU and learn on the GPU, by keeping physics and learning on the GPU and passing data directly between them (Makoviychuk et al., arXiv:2108.10470). Boston Dynamics's Spot research kit ships with a simulation environment built on Isaac Lab.
• Others, such as PyBullet and Brax, fill similar roles in research code.

The speedup matters because RL is data-hungry. When thousands of virtual robots train at once, a policy that once needed weeks can finish in minutes. That is the reason locomotion results arrived so quickly once GPU simulation matured.

The reward function: the part humans still write

The policy is learned, but the reward is written by a person, and it is where much of the engineering lives. A reward tells the agent what counts as doing well. Get it wrong and the agent will find the shortcut.

Reward style	What it looks like	Strength	Risk
Sparse	One signal at the end: the object reached the goal, yes or no	Hard to misinterpret	The agent may never stumble on success, so it learns nothing
Dense	A running score, such as distance to the goal	Gives steady guidance	The agent may chase the score instead of the goal
Shaped, multi-term	A weighted sum: forward speed, minus energy use, minus a penalty for falling, minus jerky motion	Produces smooth, natural movement	The weights are fiddly, and small changes can alter behavior

A typical locomotion reward blends several terms: reward for matching a commanded velocity, penalties for wasted energy or violent motion, and a termination when the robot falls. The balance between those terms shapes the gait that emerges. Heavy energy penalties produce smooth, efficient steps, while a weak penalty can produce a violent scramble that works in simulation and wrecks a real gearbox.

Specification gaming

DeepMind researchers use the term specification gaming for an agent that satisfies the letter of a reward while violating its intent. A walking agent rewarded only for forward speed of its torso might learn to throw itself forward in a repeating lunge. A manipulation agent rewarded for moving an object near a target might learn to push it into the target's neighborhood without ever grasping it. These are illustrations of the pattern, not reports of specific experiments. The lesson is that rewards are specifications, and a specification with a loophole will be exploited by any optimizer persistent enough to find it.

Domain randomization: teaching a policy to expect surprises

A simulator is a model, and models are wrong in small, stubborn ways. Friction is a little off, a motor responds a little slower, the camera sees a different shade of gray. A policy trained in one exact simulated world can learn to depend on its quirks, then fail on a real robot. This mismatch is the sim-to-real gap.

The standard remedy is domain randomization. Rather than training in one simulated world, the training randomizes the world: friction, mass, motor strength, sensor delay, lighting, textures. The policy cannot rely on any one setting being true, so it learns behavior that works across the range. The real world then looks like one more variation.

The idea has a clear early demonstration. In 2017, Tobin and colleagues trained object detectors purely on simulated images with non-realistic random textures, and the detectors localized real objects to about 1.5 centimeters while staying robust to distractors and partial occlusion (Tobin et al., arXiv:1703.06907).

What gets randomized	Why it matters on real hardware
Friction and surface properties	Floors, gripper pads, and objects are rarely what the simulator assumed
Mass and center of mass	Payloads and manufacturing variation change how a body balances
Motor strength and delay	Real actuators lag and saturate
Sensor noise and latency	Real sensors are noisy and late
Lighting, textures, camera pose	Vision policies must survive changes in the environment
Random pushes and disturbances	Teaches the policy to recover, not just to follow a plan

The costs of randomization are real. Too little and the policy is brittle. Too much and the policy becomes timid, because it has to hedge against conditions that never occur on the real machine. Tuning the range is part of the craft.

The Rubik's Cube hand

OpenAI's 2019 project, often called Dactyl, took domain randomization further with a method it called automatic domain randomization, which generates a distribution over randomized environments of ever-increasing difficulty. Control policies and vision-based state estimators were trained only in simulation, then deployed on a humanoid robot hand (Akkaya et al., arXiv:1910.07113).

The real-hand results were honest rather than perfect. The best policy solved a cube that needed 15 face rotations in 60 percent of trials and the hardest case, 26 face rotations, in 20 percent of trials. The authors also tested perturbations the policy had never trained on: a rubber glove, tied fingers, a blanket over the hand, and pokes with a plush giraffe and a pen. In each case the policy kept performing face rotations and flips, though the authors did not quantify performance under those conditions.

The takeaway is not that the cube was solved reliably. It is that a policy trained entirely in simulation transferred to a real, complex hand and tolerated disturbances it had not seen.

Legged locomotion: where RL has clearly worked

Walking looks easy and is not. A legged robot is an unstable machine that must make and break contact with the ground many times a second. Classical controllers handle it with models, optimization, and a lot of hand-tuning. RL offered another route: let the robot find a gait by itself.

Year	Work	What was shown	Source
2020	ANYmal in natural terrain (Lee, Hwangbo, Wellhausen, Koltun, Hutter)	A neural-network controller trained in simulation, using only proprioceptive signals, transferred zero-shot to mud, snow, rubble, and vegetation on two generations of ANYmal robots	Science Robotics 5(47), arXiv:2010.11251
2021	Massively parallel training (Rudin, Hoeller, Reist, Hutter)	Under four minutes of training for flat terrain and twenty minutes for uneven terrain, on a single workstation GPU, with thousands of simulated robots in parallel	CoRL 2021, arXiv:2109.11978
2023	ANYmal parkour (Hoeller, Rudin, Sako, Hutter)	A hierarchical, fully learned system of locomotion skills and a navigation policy, trained only in simulation, crossing consecutive obstacles at speeds up to 2 m/s	arXiv:2306.14874
2023	Humanoid soccer (Haarnoja et al.)	A 20-joint miniature humanoid trained in simulation and transferred zero-shot to hardware, walking 181 percent faster and taking 63 percent less time to get up than a scripted baseline	arXiv:2304.13653

Two details in that table deserve attention. The 2020 ANYmal work used only proprioception, the robot's internal sense of its own joints and motion, with no camera. The controller felt its way across ground it had never seen. The authors concluded that radical robustness in natural environments can be achieved by training in much simpler domains.

The 2021 paper's real contribution was speed. A controller that took hours or days to train could be rebuilt in a coffee break, which changes how researchers work. Instead of treating a policy as a rare, precious artifact, they can try ten reward variations before lunch.

What a learned controller looks like inside a product

Not every deployed robot is an RL policy from end to end. Boston Dynamics describes the change to Spot's locomotion in a way that is useful for calibrating hype. Spot's earlier controller ran many instances of model predictive control, an optimization method that plans a short sequence of motions, in parallel and picked the best one by a score. That earlier controller was reliable, with a mean time between falls of hundreds of hours, but heavy on onboard computing. The newer approach trains a neural-network policy that works alongside the existing model-based controller, which still handles what it does well. The policy was trained over more than a million simulations with varied stair dimensions, terrain roughness, and ground friction, then validated on an internal fleet logging over 2,000 hours of runtime each week. Boston Dynamics reports that Spot is less likely to fall on slick or irregular surfaces and can climb boxes more than 70 centimeters high with a learned policy paired with the model-based controller (Boston Dynamics blog).

The company also announced a Spot RL Researcher Kit, with NVIDIA and The AI Institute, that includes a joint-level control interface, a payload with an NVIDIA Jetson AGX Orin to run the policy, and an Isaac Lab-based simulation environment. It reports that its deployed Spot robots have cumulatively walked over 250,000 kilometers.

This is a more careful story than the common one. RL here is a component, trained in simulation, validated against a large fleet, and layered on a controller that already worked.

A note about Atlas

Popular accounts often credit reinforcement learning for the Atlas humanoid's 2021 parkour video, released on August 17, 2021, which showed two Atlas robots running a course of jumps, balance beams, and vaults. This article makes no claim about how those parkour behaviors were produced. Boston Dynamics retired the hydraulic Atlas on April 16, 2024, and revealed a fully electric version the next day. How its behaviors are produced is a question for the company's own technical publications.

Manipulation: harder, and more dependent on demonstrations

Moving a body is one problem. Handling objects is another, because the world offers endless shapes, weights, and surfaces, and contact is hard to simulate. A grasp can succeed or fail on a millimeter of finger position or a tiny change in friction.

For that reason, much recent progress in manipulation comes from imitation learning, where a model learns from demonstrations by people, rather than from RL alone. The two approaches blend: demonstration data gives a policy a sensible starting point, and RL can refine it against a reward.

RT-1: scale as a strategy

Google's RT-1, short for Robotics Transformer 1, took the scale route. According to the paper, it was trained on about 130,000 episodes covering more than 700 language instructions, collected over 17 months with 13 robots. It takes camera images and an instruction as input and outputs motor commands. In the paper's evaluation, RT-1 completed 97 percent of seen instructions and 76 percent of unseen ones, and held 83 percent success with distractor objects and 59 percent with changed backgrounds (arXiv:2212.06817).

The comparison with other methods in the paper shows where the gains came from:

Method	Seen tasks	Unseen tasks	Distractors	Backgrounds
RT-1	97%	76%	83%	59%
Gato	65%	52%	43%	35%
BC-Z	72%	19%	47%	41%

Source: Brohan et al., RT-1 paper (arXiv:2212.06817), evaluation results as reported. Treat the percentages as indicative, since they depend on the lab's own task set.

RT-1 is not reinforcement learning in the strict sense. It learned from human demonstrations. It belongs in this article because it set the baseline that RL-based manipulation work is compared with, and because it shows how much real-world data it takes to get a robot to handle hundreds of instructions.

RT-2: borrowing knowledge from the web

RT-2 added a different idea. The authors co-fine-tuned existing vision-language models, which are trained on web-scale images and text, on robot trajectory data, and wrote robot actions as text tokens so that one model handles both language and control. They call this class of model a vision-language-action model. The abstract reports emergent capabilities that were absent from the robot data, such as following commands that involve numbers or icons, simple reasoning about the smallest or largest object, and multi-step reasoning with chain of thought, such as choosing a rock as an improvised hammer or an energy drink for someone who is tired. The evaluation covered 6,000 trials (Brohan et al., RT-2, arXiv:2307.15818).

The idea is a form of transfer: knowledge about objects learned from internet text and images carries over to choosing and handling objects. For the broader concept, see transfer learning and why AI does not start from scratch.

Open X-Embodiment and generalist models

Different labs use different robots, which splits the data. The Open X-Embodiment collaboration pooled data from 21 institutions covering 22 robots, with 527 skills and 160,266 tasks. The authors reported that a high-capacity model trained on the pooled data, called RT-X, showed positive transfer, improving capabilities of multiple robots by using experience from other platforms (arXiv:2310.08864).

Physical Intelligence's pi-zero model followed a similar direction. Built on a pre-trained vision-language model with a flow-matching approach, it was trained on data from several dexterous platforms and demonstrated tasks such as laundry folding, table cleaning, and box assembly, according to the paper presented at RSS 2025 (arXiv:2410.24164).

The common thread is that manipulation is starting to look like language modeling: large, diverse datasets and large models, with RL often used for fine-tuning rather than as the main engine.

Contact-rich tasks and the senses robots lack

Inserting a connector, turning a cap, or fitting a part into a socket requires feeling forces, not just seeing. Simulators approximate contact with simplified models, and a policy that depends on the exact stiffness of a simulated joint may jam on a real one.

Sensing is part of the answer. A robot without force or touch sensing is working half blind in these tasks, a point developed in why robots need force sensors. Richer sensing gives a policy something to correct against when the physical world differs from the simulated one. For the larger picture of how sensing, planning, and control combine in a machine, see how modern robotics actually works and the overview of humanoid robots in 2026.

Algorithms: model-free and model-based

Robot RL tends to use a small set of algorithm families. Model-free methods learn a policy directly from experience, without learning how the world works. PPO (proximal policy optimization) and SAC (soft actor-critic) are the names that appear most often in this literature. They are comparatively simple and robust, and when simulation supplies cheap data, they dominate.

Model-based methods learn a model of the environment's dynamics and use it to plan or to generate practice. They can use data more efficiently, which matters when each sample costs real robot time, but errors in the learned model can compound.

A quick way to choose between them: when samples are cheap, as in simulation, simplicity wins. When samples are expensive, as on hardware, efficiency matters more. Most of the results above use the first approach, which fits with the field's reliance on simulation. The underlying ideas of rewards, policies, and value estimates are covered in reinforcement learning explained, and the game-playing lineage is in from AlphaGo to StarCraft.

What still goes wrong

Reality has long tails. A policy that works 99 times in 100 may be acceptable in a lab and unacceptable in a warehouse. Validating rare failures takes fleet-scale testing, which is what Boston Dynamics describes doing with Spot.

Rewards are fragile. As the table above shows, small changes to reward weights change behavior. Debugging a reward often takes longer than training.

Simulators are expensive to make faithful. Contact, deformable objects, cables, liquids, and fabric remain hard to model. Domain randomization handles some mismatch, but it cannot cover physics that the simulator cannot represent at all.

Hardware limits what policies may do. A policy found in simulation may command torques that heat or damage a real motor. Layering a learned policy on a controller with built-in limits, as in the Spot example, is one answer.

Generalization is narrower than demos suggest. RT-1's 19 percent for BC-Z on unseen tasks versus 76 percent for RT-1 shows that generalization is possible, but the 59 percent score on changed backgrounds shows how much is left.

The compute side is not free either. Training at this scale depends on GPUs, a point covered in how GPUs work and why AI needs them. And the economic backdrop, with factory robots already common, is laid out in countries with the most robots in manufacturing.

Common misunderstandings

"RL robots learn from scratch in the real world." Almost none do. Locomotion policies are trained mostly in simulation, and manipulation systems lean on demonstration data.

"RT-2 is reinforcement learning." It is a vision-language-action model trained on robot trajectories and web data. It shows that language models can help robots choose actions, not that RL solved manipulation.

"A policy that works in simulation will work on the robot." That is the sim-to-real gap, and domain randomization exists because the answer is often no.

"End to end means no engineering." Reward design, simulator tuning, safety limits, and fleet validation are all engineering, and the Spot example shows learned and classical control working together.

"Humanoid demos prove general-purpose robots are near." A demo shows that something can work. It does not show how often it works, in how many settings, or at what cost. Reliability data is what separates the two.

FAQ

Q: What reward function do RL robotics systems use to train locomotion or manipulation? A: Rewards are written by people and tuned by trial. Locomotion rewards typically combine forward or commanded velocity with penalties for wasted energy, harsh motion, and falling. Manipulation rewards are often sparse, such as whether an object reached the target, sometimes with dense shaping terms like distance to the goal. Poorly specified rewards invite specification gaming, where the agent exploits a loophole instead of doing the task.

Q: How long does it take to train a robot RL policy in simulation? A: It depends on the task. For a quadruped learning to walk, a 2021 ETH Zurich study reported under four minutes on flat terrain and twenty on uneven terrain, using a single workstation GPU. Complex manipulation takes far longer: OpenAI's Rubik's Cube hand policy used roughly 13 thousand years of cumulative simulated experience, spread across many parallel simulations.

Q: What is the difference between model-free and model-based RL for robotics? A: Model-free methods, such as PPO and SAC, learn a policy directly from experience. They are simpler and tend to be robust, and they dominate when simulation supplies cheap data. Model-based methods learn a model of the world's dynamics and plan with it, which can save samples but risks compounding model errors. Hardware, where samples are costly, favors efficiency.

Q: Can RL robots handle tasks they were never trained on? A: To a limited degree. RT-1 completed 76 percent of unseen instructions in its evaluation, compared with 97 percent for seen ones. RT-2 reported emergent abilities, such as reasoning about which object could work as a hammer, by borrowing knowledge from web-trained vision-language models. Neither amounts to reliable handling of arbitrary new tasks, and both rely on large demonstration datasets as much as on RL.

Q: What simulation environments are used for robot RL research? A: MuJoCo, acquired by Google DeepMind in October 2021 and open-sourced in May 2022, is widely used for research. NVIDIA's Isaac Gym and Isaac Lab run physics on the GPU for massively parallel training, and the 2021 Isaac Gym paper reported two to three orders of magnitude improvement over conventional CPU-simulation setups. PyBullet and Brax are also common in research code.

Q: Is the Boston Dynamics Atlas robot controlled by reinforcement learning? A: That is not established here. Boston Dynamics has published that Spot's locomotion now uses a neural-network policy trained with RL alongside its model-based controller, but this article makes no claim about how Atlas's 2021 parkour was produced. The company's own technical publications are the place to look.

Q: Why not just train robots in the real world? A: Real robots wear out, need human resets, and run in real time. RT-1's data collection, 130,000 demonstration episodes over 17 months with 13 robots, shows the cost even for imitation learning. Simulation removes those limits for the bulk of training, with real-world testing reserved for validation and fine-tuning.

The takeaway

Reinforcement learning for robots works best where a simulator can supply huge amounts of cheap experience and where the physics is within reach of the model. That describes legged locomotion, and the results there are real: policies trained in minutes that handle mud, rubble, stairs, and obstacles on hardware. It describes manipulation less well, which is why the most visible progress there comes from large demonstration datasets and web-trained models, with RL as one ingredient among several.

The pattern across the evidence is layering. Boston Dynamics keeps a model-based controller beside a learned policy. RT-2 starts from a language model. Dactyl solved a cube 20 to 60 percent of the time, depending on difficulty, and was honest about it. The robots that matter in the next few years will probably be the ones whose makers can say how often they fail, not the ones that look best in a short video.

Sources

• Akkaya, I. et al. (2019), Solving Rubik's Cube with a Robot Hand, arXiv:1910.07113
• Tobin, J. et al. (2017), Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World, arXiv:1703.06907
• Rudin, N., Hoeller, D., Reist, P., and Hutter, M. (2021), Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning, arXiv:2109.11978
• Lee, J. et al. (2020), Learning quadrupedal locomotion over challenging terrain, Science Robotics, arXiv:2010.11251
• Brohan, A. et al. (2022), RT-1, arXiv:2212.06817; (2023), RT-2, arXiv:2307.15818
• Open X-Embodiment Collaboration (2023), arXiv:2310.08864
• Boston Dynamics, Starting on the right foot with reinforcement learning

---

Read next:

• Edge Computing and Federated Learning: Training AI Without Exposing Your Data
• How Modern Robotics Actually Works
• Machine Learning in Cybersecurity: How AI Detects Intrusions Before Humans Can