How Do Self-Driving Cars Actually Work?

A self-driving car is one of the most complex robots ever built. It must perceive its environment in real time, predict what other road users will do, decide what to do itself, and execute physical actions, all while moving at speed with human lives at stake. Companies have spent well over a decade and many billions of dollars on the problem, and the results are uneven: driverless taxis now carry paying passengers in several American cities, while the car in most driveways still needs a human watching the road. This article explains how the technology actually works, what the published safety data does and does not show, and why "drive anywhere, anytime" autonomy remains much harder than it looks.

Key takeaways

• Self-driving systems combine cameras, radar, lidar, ultrasonic sensors, satellite positioning and inertial sensors to perceive the world.
• The software stack has five broad jobs: perception, localisation, prediction, planning and control.
• The industry standard SAE J3016 defines Levels 0 to 5. Levels 0–2 support a human driver; Levels 3–5 are "automated driving" where the system drives within its design limits.
• Most cars sold today offer Level 2 features, where the human remains responsible. Driverless Level 4 robotaxis operate only inside defined service areas.
• Company-published safety data is encouraging but comes with caveats about comparison baselines, and edge cases such as unusual road works, emergency scenes and heavy weather remain the hardest problems.

The automation levels (SAE J3016)

The levels most people quote come from SAE International's standard J3016, whose current revision dates from April 2021. The key idea is not how clever the car is but who is responsible for driving at a given moment, and under what conditions. Those conditions are called the operational design domain (ODD): the roads, speeds, weather, times of day and geographic area the system was designed to handle.

Level	SAE name	Who drives	Typical example
0	No driving automation	Human, with warnings at most	Lane-departure warning, automatic emergency braking
1	Driver assistance	Human, with steering or speed support	Adaptive cruise control
2	Partial driving automation	Human, with steering and speed support; must supervise constantly	Tesla Autopilot / FSD (Supervised), GM Super Cruise, Ford BlueCruise
3	Conditional driving automation	System, within its ODD; human must take over when asked	Mercedes-Benz Drive Pilot on approved motorways
4	High driving automation	System, within its ODD; no human fallback needed	Waymo robotaxis in their service areas
5	Full driving automation	System, everywhere a human could drive	Does not exist

Two points are easy to miss. First, the jump from Level 2 to Level 3 is a jump in legal and moral responsibility, not merely in capability. A Level 2 system can be very capable and still require the driver to watch the road at every moment, because the manufacturer has not designed or validated it to cope alone. Second, Level 4 is not "Level 5 with a few gaps". A Level 4 vehicle is fully driverless, but only within a carefully bounded domain. Outside it, the vehicle must refuse to drive or pull over safely.

Mercedes-Benz's Drive Pilot is the best-known Level 3 example. The German Federal Motor Transport Authority (KBA) approved an upgraded version in December 2024 that can operate in flowing traffic up to 95 km/h on German motorways, up from an earlier limit of about 60 km/h, and California and Nevada have certified the system since 2023 at the lower speed. While it is engaged, the car rather than the driver is responsible, but the driver must be ready to take back control within a few seconds when prompted.

How a self-driving car perceives the world

No single sensor is good at everything, so autonomous vehicles combine several types whose strengths and weaknesses complement each other. Combining them is called sensor fusion.

Sensor	What it measures	Strengths	Weaknesses
Camera	Colour images (2D)	Reads signs, lights, lane paint; high resolution; cheap	Struggles with glare, darkness, heavy rain; depth must be inferred
Radar	Range and radial speed via radio echoes	Works in fog, rain, darkness; measures speed directly via Doppler	Low angular resolution; metal clutter confuses it
Lidar	3D point cloud via laser time of flight	Centimetre-scale geometry regardless of lighting	Cost; degraded by heavy rain, snow and spray; no colour
Ultrasonic	Distance to nearby objects	Cheap, good for parking	Range of only a few metres
GNSS + IMU	Global position; acceleration and rotation	Absolute position plus smooth motion tracking	Satellite signals blocked in tunnels and city canyons

Cameras

Cameras provide the richest information per dollar. They are the only sensor that can read the text on a sign, tell a red traffic light from a green one, or notice that a police officer is waving traffic through. Modern systems run several cameras covering 360 degrees. Their main limitation is that a single image has no direct depth information; distance must be estimated from stereo pairs, motion between frames or learned cues, and that estimation degrades in poor light and bad weather. The basic optics are the same as in any digital camera; the history of the camera from camera obscura to digital sensors explains how image sensors turn light into numbers.

Radar

Radar sends out radio waves and listens for echoes. The time delay gives range and the Doppler shift gives the object's speed toward or away from the car, measured directly rather than inferred. Radar keeps working in fog, darkness and spray, which is why it has been the backbone of adaptive cruise control for decades. Traditional automotive radar, however, has poor angular resolution: it knows something is 60 metres ahead and closing at 5 m/s, but not much about its shape. Newer "imaging" radars with many more antennas narrow that gap. The underlying principle dates back to the Second World War, as covered in the story of radar.

Lidar

Lidar (light detection and ranging) fires laser pulses and times how long they take to bounce back. Doing that hundreds of thousands of times per second builds a 3D point cloud of the surroundings, accurate to a few centimetres. Lidar sees the shape of a pedestrian or a fallen ladder whether it is noon or midnight. It has historically been bulky and expensive, though prices have fallen sharply as solid-state designs mature. For the physics of time-of-flight measurement and how scanners are built, see how LiDAR works.

Ultrasonic sensors, GNSS and inertial sensors

Ultrasonic sensors in the bumpers measure short distances and are mostly useful for parking. Satellite positioning (GNSS, of which GPS is one system) gives a global position accurate to a few metres; an inertial measurement unit (IMU) tracks acceleration and rotation hundreds of times per second, filling the gaps when satellite signals drop out.

High-definition maps

Many autonomous systems also carry detailed prior maps that record lane geometry, curb heights, traffic-light positions and speed limits to a level of detail far beyond a navigation app. The map acts as a kind of extra sensor: the car can compare what it sees against what it expects, and focus its attention on what has changed. The trade-off is cost and maintenance; every road must be mapped and the map must be kept current.

The software stack, layer by layer

Raw sensor data is only the beginning. A typical autonomous vehicle runs a pipeline of software modules that repeats many times per second.

1. Perception

Perception turns pixels, radar returns and point clouds into a list of objects with positions, sizes, velocities and categories: "pedestrian, 8 metres ahead, walking left at 1.4 m/s; car approaching from the right at 30 km/h; traffic light green." Most of this is now done with deep neural networks trained on very large labelled datasets. How neural networks work covers the underlying architecture; in a car, networks perform object detection, segmentation of drivable space, and tracking of objects from one frame to the next.

Fusion can happen early (combining raw data before detection), late (each sensor detects objects separately and the results are merged) or somewhere in between. A popular modern representation is the bird's-eye-view grid, in which camera features are projected onto a top-down map of the area around the car.

2. Localisation

The car must know where it is to within a few centimetres, not a few metres, to stay in its lane. GNSS alone cannot do that reliably. Instead the system matches live lidar or camera data against its prior map, blending in IMU and wheel-speed data with filtering techniques such as the Kalman filter. Systems that work without detailed maps must instead infer lane positions from what the cameras see in real time.

3. Prediction

Knowing where everything is now is not enough; the car must estimate where everything will be over the next several seconds. Will the cyclist swerve around the parked van? Is the pedestrian at the curb about to step out? Modern prediction models output several possible futures for each agent, each with a probability, and they take interactions into account: the car ahead is likely to brake because the car ahead of it is braking.

4. Planning

The planner decides what the car itself should do. At the top level it handles routing; in the middle it chooses behaviours such as following, overtaking, yielding or changing lanes; at the bottom it computes a precise trajectory for the next few seconds. Classical planners score many candidate trajectories with a cost function that penalises collision risk, rule violations, harsh braking and discomfort, then pick the lowest-cost option. A growing share of planning is now learned from data rather than hand-written.

5. Control

Control converts the chosen trajectory into steering angle, throttle and brake commands, correcting continuously for the gap between where the car is and where it should be. Techniques such as model predictive control are common. Control also includes the safety layers underneath everything: redundant braking and steering hardware, backup power, and a "minimal risk manoeuvre" that brings the vehicle to a safe stop if something fails.

Modular versus end-to-end

The five-layer picture describes a modular stack, in which engineers can inspect the output of each stage. Several companies are moving toward end-to-end approaches, where a single large neural network maps sensor input more or less directly to driving output, trained on huge amounts of recorded human driving. End-to-end systems can handle subtle situations that are hard to write rules for, but they are harder to inspect and verify. In practice many teams use hybrids: learned components for perception, prediction and planning, wrapped in rule-based safety checks.

Two strategies: vision-first versus full sensor suite

The industry has split broadly into two philosophies.

Tesla: cameras and fleet data

Tesla relies on cameras and neural networks, having dropped radar from new vehicles and never adopted lidar. Its argument is that humans drive with two eyes and a brain, so a sufficiently good vision system should be able to as well, and that cameras are cheap enough to fit to millions of customer cars, which then supply training data. The company's consumer product is now called Full Self-Driving (Supervised), a Level 2 system in which the driver remains responsible at all times. Tesla began a small paid robotaxi service in Austin, Texas, on 22 June 2025 with a Tesla employee in the front passenger seat as a safety monitor; some later reports describe rides without an in-car monitor but with chase vehicles, and the service remains small. The US National Highway Traffic Safety Administration (NHTSA) has contacted Tesla about incidents in that service.

Pros: low hardware cost; one system that can in principle run anywhere there are roads; very large training fleet. Cons: cameras lose information in poor visibility; depth is inferred rather than measured; independent safety data for unsupervised operation is still limited.

Waymo: cameras, lidar, radar and maps

Waymo, which grew out of Google's self-driving project, uses all three main sensor types plus detailed maps, and operates fully driverless (Level 4) robotaxis. As of mid-2026 its fully driverless service areas included the San Francisco Bay Area, Phoenix, Los Angeles, Austin, Atlanta and Miami, among others, with further cities being added.

Pros: redundant sensing; measured rather than inferred depth; a large record of driverless miles. Cons: expensive vehicles; each new city requires mapping and validation; growth is slower.

Others

Other players sit between those poles. Zoox (owned by Amazon) builds a purpose-designed vehicle without a steering wheel. Mobileye supplies camera-based driver assistance to many carmakers and is developing higher-level systems. In China, Baidu's Apollo Go and Pony.ai run robotaxi services in several cities. Not every effort survives: General Motors announced in December 2024 that it would stop funding the robotaxi business of its Cruise unit, more than a year after a Cruise vehicle dragged a pedestrian in San Francisco and California suspended its driverless permit.

What the safety data shows, and what it does not

"Are self-driving cars safer than humans?" is the question everyone asks, and the honest answer is: in some well-defined settings, the evidence is promising, but the comparison is harder than it looks.

Company data: Waymo

Waymo publishes a running comparison of its rider-only (no human driver) miles against human-driver benchmarks for the same cities and road types. Through June 2026 it reported about 271 million rider-only miles. Against its human benchmark, it reports:

Outcome (Waymo vs human benchmark, same cities)	Reported reduction
Crashes with serious injury or worse	95% fewer
Crashes with an airbag deployment in any vehicle	82% fewer
Crashes causing any injury	82% fewer
Pedestrian crashes with injuries	93% fewer

Source: Waymo Safety Impact hub, data through June 2026.

These are large differences, and the dataset is now big enough that they are unlikely to be pure chance. Caveats still apply. The data is published by the company itself. Human benchmarks are built from police-reported and insurance data that undercount minor crashes, and adjusting them for the same streets, times and conditions involves modelling choices. Robotaxis also avoid some situations entirely, such as motorways in some markets or certain weather.

Regulatory data: NHTSA's crash reporting order

In the United States, NHTSA's Standing General Order 2021-01, first issued in June 2021 and amended in 2021, 2023 and 2025, requires manufacturers and operators to report certain crashes involving automated driving systems (Level 3 and above) and Level 2 driver-assistance systems. For automated systems, a crash must be reported if the system was in use within 30 seconds of it and it met triggers such as an injury requiring hospital treatment, a fatality, a vulnerable road user being struck, an airbag deployment or a vehicle being towed. Level 2 reporting triggers are narrower. The amendment that took effect in June 2025 relaxed some of the timelines.

NHTSA cautions that these reports are not normalised by miles driven and should not on their own be used to compare companies. The value of the order is transparency: it shows regulators and the public what kinds of crashes occur, which feeds investigations and recalls.

Why Level 2 is a special problem

Level 2 systems ask humans to supervise automation that works well most of the time. Research on automation in aviation and elsewhere has long found that people are poor at sustained passive monitoring and slow to re-engage when something goes wrong. That is why driver-monitoring cameras, which check that the driver is looking at the road, have become a central safety feature, and why marketing names that imply more capability than the system has attract regulatory criticism.

Why self-driving is so hard

The long tail

Most driving is routine: marked lanes, clear weather, predictable traffic. But the list of unusual situations is effectively endless: a mattress falling off a truck, a traffic officer overriding a green light, a child chasing a ball, a flooded underpass, a funeral procession, a horse on a rural road. Each is rare; together they are common. Handling them requires both enormous amounts of diverse data and a system that behaves sensibly when it meets something it has never seen.

Generalisation

A system tuned in sunny Phoenix may struggle with snow in Minnesota, left-hand traffic in Tokyo or informal road use in a city where lanes are suggestions. This is one reason Level 4 services expand city by city.

Predicting humans

People communicate on the road with eye contact, hand waves and small hints such as a car edging forward. Reading those cues, and knowing when other drivers expect you to assert yourself at a four-way stop, is difficult to learn and varies by culture.

Adversarial and degraded conditions

Glare at sunset, a dirty camera lens, a sticker on a stop sign or spray from a lorry can confuse perception. Robust systems need redundancy, so that no single sensor failure leads to a dangerous decision, and they must recognise when their own confidence has dropped.

Verification

Traditional safety engineering checks that software meets a written specification. A neural network trained on millions of examples does not have a line-by-line specification. Developers instead combine simulation (replaying and varying billions of virtual miles), closed-course testing, staged public deployment with safety drivers, and statistical arguments based on miles driven. Standards such as ISO 26262 (functional safety) and ISO 21448 (safety of the intended functionality) provide frameworks, but agreeing on "how safe is safe enough" is partly a social and legal question, not only a technical one.

Remote assistance

Driverless fleets are supported by human operators in remote centres who can give guidance when a vehicle is unsure, for example by confirming that it may pass a blocked lane. They generally do not steer the car with a joystick; the vehicle remains responsible for driving. How many vehicles each operator can support is an important factor in the economics of robotaxis.

Where autonomy works today

Robotaxis

Fully driverless paid rides operate in a number of US cities through Waymo and in several Chinese cities through companies such as Baidu and Pony.ai, with other services in trial phases.

Motorways

Many carmakers sell Level 2 motorway assistance, some of it allowing hands-off driving on mapped highways with camera-based driver monitoring. Level 3 operation, where the car is responsible, is available only in a few systems under tight conditions.

Long-haul trucking

Motorway freight is an attractive target: relatively simple roads, long distances and a shortage of drivers. Aurora Innovation announced on 1 May 2025 that it had begun commercial driverless deliveries with heavy-duty trucks between Dallas and Houston in Texas, after years of testing with safety drivers on Interstate 45.

Industrial and off-road settings

Mines, ports, farms and warehouses have used automated vehicles for years, because the environment is controlled and access can be restricted. These are among the most commercially mature uses, even if they rarely make headlines. Related approaches drive autonomous navigation on Mars rovers, where communication delays force the vehicle to make many decisions on its own.

Common misconceptions

"Fully self-driving cars are about to be everywhere"

Driverless robotaxis are real today, but only inside defined service areas. A Level 5 car that can go anywhere a human can, in any weather, does not exist, and no credible developer has a public date for one.

"If it's called self-driving, I don't need to watch the road"

Product names are not SAE levels. If a system is Level 2, the driver is responsible at all times, whatever the marketing says. The owner's manual and the system's own warnings are the authority.

"Lidar is always better than cameras" (or the reverse)

Each sensor sees different things. Lidar gives direct geometry; cameras give meaning; radar gives speed and works in fog. The debate is about cost, scale and how much redundancy is needed, not about one sensor being universally superior.

"Self-driving cars don't crash"

They do crash, and some crashes are caused by other road users. The relevant question is how often serious crashes happen per mile compared with human drivers in the same conditions, and which new failure modes the technology introduces.

"Better AI alone will solve it"

Regulation, liability, insurance, public acceptance, remote support, maintenance and the economics of expensive vehicles all shape deployment as much as algorithms do. For the batteries and drivetrains underneath most of these vehicles, see how electric cars work.

Frequently asked questions

Q: When will I be able to buy a car that drives itself everywhere? A: Nobody knows. Personally owned cars with Level 3 features exist in limited markets and conditions, but a Level 5 car that can drive anywhere without supervision is not on sale and has no credible public launch date.

Q: Are self-driving cars safer than human drivers? A: In some defined settings the evidence is encouraging. Waymo reports large reductions in injury crashes against human benchmarks in its operating cities, but that data is company-published and depends on how the benchmark is built. Results do not automatically transfer to other systems or places.

Q: What is the difference between Level 2 and Level 3? A: At Level 2 the human is always driving and must supervise the system. At Level 3 the system drives within its design conditions and the human may look away, but must take back control when asked.

Q: Can self-driving cars handle rain or snow? A: Heavy rain, snow and spray degrade cameras and lidar, and snow can hide lane markings. Radar copes better. Most driverless services restrict operations in severe weather, and expanding the weather range is an active area of work.

Q: Who is liable when an automated car crashes? A: It depends on the level and on local law. With Level 2 systems the driver is generally responsible. When a Level 3 or Level 4 system is driving within its design limits, responsibility shifts substantially toward the manufacturer or operator, and several countries have passed laws to clarify this.

Q: Do self-driving cars need the internet to work? A: No. Driving decisions are made by computers on board, because a network delay could be dangerous. Connectivity is used for map updates, fleet management, software updates and remote assistance.

Q: Will self-driving vehicles replace professional drivers? A: Long-haul trucking, taxis and delivery are the most exposed jobs, but the pace depends on safety validation, regulation and cost. For a wider view of how automation reshapes work, see how AI will change jobs.

The bottom line

Self-driving cars work by layering many imperfect sensors and many learned and hand-built software modules into a system that is more reliable than any single part. The technology is real and improving: driverless robotaxis carry passengers in several cities, driverless lorries haul freight on Texas motorways, and the best published safety data is encouraging. But autonomy arrives domain by domain, city by city and condition by condition, not all at once. The useful questions to ask of any system are always the same: what level is it, what is its operational design domain, and who is responsible when it is engaged?

Sources

• Waymo Safety Impact hub, rider-only miles and crash-rate comparisons, data through June 2026.
• NHTSA, Standing General Order 2021-01 on crash reporting (Third Amended order, effective June 2025).
• NPR: GM to retreat from robotaxis and stop funding its Cruise autonomous vehicle unit (December 2024).
• Business Wire: Aurora Begins Commercial Driverless Trucking in Texas (1 May 2025).
• Automotive World: Mercedes-Benz approved for 95 km/h Level 3 driving in Germany (December 2024).
• SAE International, J3016 Taxonomy and Definitions for Terms Related to Driving Automation Systems, revision April 2021.

Related Reading

• What Are Deepfakes — and How Do They Actually Work?
• What Is Neuralink? How Brain-Computer Interfaces Actually Work
• How Drones Actually Work — and Where They're Going
• How Do Quantum Computers Actually Work? Beyond the Hype
• What Are AI Agents — and How Do They Actually Work?