How Gradient Descent Finds Better Answers

A neural network with a billion parameters has no algebraic formula anyone could solve directly to find the single best value for every parameter at once. Gradient descent sidesteps that impossibility with a strategy that works even blindfolded: feel the local slope beneath your feet, and step downhill. Repeat that step thousands or millions of times, and the model's error, measured by a loss function, shrinks toward a low point, all without ever surveying the entire landscape at once.

That single idea, formalized nearly two centuries ago for an entirely different purpose, is now the mechanism that trains almost every deep learning system in production, from spam filters to large language models. Understanding how it actually works, and where it breaks down, explains a lot about why modern AI needs so much compute, why training sometimes stalls, and why a handful of optimizer tweaks published over the past few decades made deep learning practically trainable at all.

What a loss function actually measures

Every trainable machine learning model has a loss function (also called a cost function or objective function), a single number that scores how wrong the model's current predictions are compared to the true answers in its training data. A high loss value means the model's predictions are far off; a low loss value means they're close. Training a model is, mathematically, nothing more than searching for the parameter values, weights in a neural network, coefficients in a linear regression, that make this one number as small as possible.

The reason this search is hard has nothing to do with the concept being complicated and everything to do with scale. A simple linear regression with two parameters has a loss surface you could sketch on paper, a bowl shape you could stare at and immediately see the bottom of. A modern transformer language model has hundreds of billions of parameters, which means its loss function lives in a space with hundreds of billions of dimensions. There is no way to visualize that surface, no way to solve for its minimum algebraically, and no way to brute-force check every possible combination of parameter values, the combinatorics are astronomically larger than the number of atoms in the observable universe.

Gradient descent gets around this by never trying to see the whole surface. At any given point, it only asks one question: which direction, from right here, reduces the loss fastest? It answers that question using calculus, takes a small step in that direction, and asks the question again from the new position. That loop, evaluate the local slope, step, repeat, is the entire algorithm. Everything else, learning rates, momentum, batching, adaptive per-parameter step sizes, is refinement layered on top of that one core loop to make it faster, more stable, or better suited to specific kinds of problems.

Augustin-Louis Cauchy, the French mathematician, is generally credited with the first written description of a steepest-descent method, which he presented to the French Academy of Sciences in 1847 while working on systems of equations related to astronomical orbit calculations, well over a century before anything resembling a modern neural network existed. The core mathematical idea, follow the steepest downhill direction to minimize a function, has not changed since. What has changed is the scale of the problems it now gets thrown at, and the engineering built around it to survive that scale.

What a gradient actually tells the algorithm

The word "gradient" has a precise mathematical meaning that's worth unpacking, because the whole name of the technique depends on it. The gradient of a function at a given point is a vector, an arrow with both a direction and a magnitude, that points toward the direction of steepest increase of that function at that exact point. For a loss function with many parameters, the gradient has one component per parameter: each component is the partial derivative of the loss with respect to that one parameter, holding everything else fixed, and it tells you how sensitive the loss is to small changes in that specific value.

Since the entire goal of training is to make the loss smaller, not larger, gradient descent moves in the exact opposite direction of the gradient at each step. That's the "descent" half of the name: gradient identifies which way is steepest uphill, and the algorithm deliberately goes the other way. The update rule, stripped of implementation detail, is almost embarrassingly simple: new parameter value equals old parameter value, minus a small factor called the learning rate, times the gradient at the current position.

A useful mental picture is a topographic map versus an actual hike. If you had a complete topographic map of a mountain range, you could plan the single most efficient path to the lowest valley in advance. Gradient descent has no map. It only has a compass that, at any given spot, points toward the steepest uphill direction, and its whole strategy is "turn the compass around and take a step." For low-dimensional problems this seems almost too primitive to work well, but for loss surfaces with billions of dimensions, having any reliable local signal at all, computed cheaply at each step, turns out to be enormously powerful, and it's essentially the only strategy that scales.

There's a second important property worth naming here: the gradient only ever describes the local neighborhood around the current parameter values. It says nothing about what the loss surface looks like ten steps away, and it says nothing about the entire global shape of the landscape. That's simultaneously the algorithm's greatest strength, it needs almost no advance knowledge of the function it's minimizing, and its most consistent source of failure modes, which shows up repeatedly in the sections below.

Why step size is the single trickiest setting to get right

The learning rate is the factor that controls how large a step the algorithm takes at each iteration, and in practice it's one of the most consequential numbers in the entire training process. Getting it wrong produces two very different, and roughly opposite, failure modes, and both are common enough in real training runs that most practitioners have hit each one personally.

A learning rate that's too small makes descent crawl. Every step nudges the parameters by a tiny amount, so the loss does eventually decrease, but so slowly that reaching a good minimum could take an impractical number of iterations, sometimes so many that a training run would need to continue for weeks or months longer than is computationally or financially feasible. Worse, an extremely small learning rate can make training look like it has stalled entirely, since the loss curve on a graph appears nearly flat even though the algorithm is technically still making (glacial) progress.

A learning rate that's too large produces the opposite failure. Instead of stepping carefully downhill, the algorithm overshoots the bottom of the valley entirely, landing on the far slope, sometimes higher up than where it started. Repeated overshooting causes the loss to oscillate wildly, bouncing back and forth across the valley instead of settling into it, and in the worst cases the parameter values diverge, growing without bound as each oversized step lands the model further from any reasonable solution rather than closer.

Learning rate	What happens during training	What it looks like on a loss curve
Too small	Extremely slow, often impractically slow, convergence	Nearly flat, barely trending down
Well-tuned	Steady, efficient progress toward a low loss value	Smooth downward curve, occasional small noise
Too large	Overshooting, oscillation, sometimes outright divergence	Jagged, spiking, or trending upward
Scheduled/decaying	Large early steps for fast initial progress, smaller later steps for fine convergence	Fast early drop, smooth flattening near the end

A learning rate that's off by an order of magnitude in either direction can be the difference between a model that trains successfully and one that never converges at all.

In practice, most modern training runs don't use a single fixed learning rate for the entire process. Learning rate schedules start with a comparatively larger step size, to make fast progress while far from any minimum, and gradually shrink it as training proceeds, allowing the algorithm to settle precisely into a low-loss region without overshooting once it's close. Common schedule shapes include step decay, cosine annealing, and a brief initial "warmup" period where the learning rate ramps up from near zero before the main schedule takes over, a technique widely used in training large transformer models to avoid instability in the earliest steps.

Why real training almost never uses the full dataset per step

The textbook description of gradient descent calculates the exact gradient using every single example in the training set before making even one parameter update. This approach, called batch gradient descent, gives the most accurate possible estimate of the true downhill direction at each step, because it's averaging the error signal over the entire dataset rather than a sample of it. But for datasets with millions or billions of examples, which describes essentially every dataset used to train a modern deep learning model, this is computationally impractical: a single parameter update would require a full pass over the whole dataset, and training could involve thousands of such updates.

Stochastic gradient descent (SGD) solves this by estimating the gradient using just one randomly chosen training example, or more commonly in modern practice, a small randomly sampled subset called a mini-batch, typically somewhere between 32 and a few thousand examples depending on the model and available hardware. Each mini-batch produces a noisy, imperfect estimate of the true gradient across the whole dataset, but that noisy estimate is dramatically cheaper to compute, which means far more parameter updates can happen in the same amount of wall-clock time.

Herbert Robbins and Sutton Monro formalized the underlying mathematical theory of stochastic approximation methods in a 1951 paper published in the Annals of Mathematical Statistics, laying groundwork that predates deep learning by decades but that modern mini-batch training still rests on. The practical insight their work supports is that a sequence of noisy, cheap gradient estimates can converge just as reliably, and often faster in real wall-clock time, than a sequence of expensive, precise ones, provided the noise is unbiased and the step sizes are chosen appropriately.

Mini-batch gradient descent, the middle ground between full-batch precision and single-example stochastic descent, is what nearly all production deep learning actually uses today. It balances two competing pressures: batches large enough to give a reasonably stable, low-noise gradient estimate, and batches small enough to fit in GPU or TPU memory and to allow frequent, fast parameter updates. Batch size interacts with the learning rate in ways that took the field years to characterize properly; larger batches generally tolerate, and often benefit from, proportionally larger learning rates, a relationship explored in detail by Goyal et al. in their widely cited 2017 paper on training ImageNet models in one hour using large-batch SGD.

How mini-batch noise can actually help

Counterintuitively, the noise introduced by using small, randomly sampled mini-batches instead of the full dataset isn't purely a downside. That noise acts like a small random jitter added to each step, which can help the optimizer bounce out of shallow local minima or narrow, sharp regions of the loss landscape that generalize poorly, nudging it toward broader, flatter minima that tend to perform better on data the model hasn't seen during training.

Why the path downhill isn't a straight line

Real loss landscapes for complex, non-convex models like deep neural networks are nothing like the smooth, single-bottomed bowl used in introductory explanations. They contain local minima, points that look like the lowest point of the surface within their immediate neighborhood but aren't the true lowest point across the entire landscape, and saddle points, locations where the slope flattens out or even curves upward in some directions while still descending in others, which can slow an optimizer to a crawl even though it hasn't actually reached any kind of minimum.

For decades, local minima were treated as the primary theoretical obstacle to training deep networks successfully, the worry being that gradient descent would frequently get trapped in a mediocre valley far from the best possible solution. More recent theoretical and empirical work on high-dimensional, non-convex loss landscapes, including analysis by Dauphin et al. presented at NeurIPS in 2014, has argued that in the extremely high-dimensional parameter spaces typical of large neural networks, saddle points are actually far more numerous and more troublesome in practice than true local minima, since it becomes statistically unlikely for every one of billions of dimensions to be curving upward simultaneously at any given point.

Momentum-based methods address this directly by giving the optimizer a form of inertia. Instead of responding purely to the current step's gradient, a momentum term accumulates a running average of past gradients, so the parameter update carries some of its previous direction forward, much like a ball rolling downhill keeps rolling through small bumps and shallow flat patches rather than stopping dead at every minor irregularity in the terrain. This helps the algorithm power through flat or slightly uphill saddle regions that would otherwise stall a plain gradient descent step.

Deep learning practitioners often treat local minima as a solved non-problem for large networks, the actual bottleneck in practice tends to be saddle points and poorly conditioned regions of the loss surface, not getting permanently trapped in a bad valley.

Adam (short for Adaptive Moment Estimation), introduced by Diederik Kingma and Jimmy Ba in a 2014 paper first presented at the International Conference on Learning Representations, combines momentum with a second technique: maintaining a separate, adaptive learning rate for every individual parameter, based on estimates of both the first moment (a running average of the gradient itself) and the second moment (a running average of the squared gradient) for that parameter. In practice, this means parameters that have historically received large, volatile gradient signals get automatically smaller effective step sizes, while parameters with small, stable gradients get relatively larger ones, without any manual per-parameter tuning required.

Comparing the main gradient descent variants

Choosing among these variants in practice usually comes down to dataset size, available compute, and how much manual tuning a team is willing to do. The table below summarizes the practical tradeoffs among the approaches described above.

Variant	Gradient computed from	Update frequency	Main strength	Main weakness
Batch gradient descent	Entire training dataset	Once per full pass (epoch)	Most accurate, lowest-noise gradient direction	Impractically slow and memory-heavy on large datasets
Stochastic gradient descent (SGD)	One randomly chosen example	Every single example	Fast, frequent updates; cheap per step	Very noisy gradient estimate, unstable convergence
Mini-batch gradient descent	Small random subset (e.g., 32–4096 examples)	Every mini-batch	Practical middle ground; hardware-friendly	Requires tuning batch size alongside learning rate
SGD with momentum	Mini-batch, plus running average of past gradients	Every mini-batch	Powers through saddle points and shallow noise	Adds a tunable momentum hyperparameter
Adam	Mini-batch, with per-parameter adaptive rates	Every mini-batch	Fast convergence with little manual tuning; default choice for many architectures	Can sometimes generalize slightly worse than well-tuned SGD on certain tasks

Mini-batch descent with an adaptive optimizer like Adam is the default combination for most modern deep learning workloads, but plain SGD with momentum remains common, and sometimes preferred, for certain computer vision architectures where its generalization behavior has been shown to be favorable.

Why this one idea powers nearly all of modern AI

Gradient descent by itself only tells the algorithm how to update parameters once it already knows the gradient. For a multi-layer neural network, actually computing that gradient efficiently, for every single parameter, across every layer, is a separate and nontrivial problem, and that's the job of backpropagation. Backpropagation applies the chain rule of calculus repeatedly, propagating the error signal backward from the output layer through each preceding layer, so that the gradient with respect to every weight in the network can be computed in a single backward pass rather than requiring a separate, expensive calculation for each individual parameter.

The algorithm was popularized in its modern neural network form by David Rumelhart, Geoffrey Hinton, and Ronald Williams in a landmark 1986 paper published in Nature, "Learning representations by back-propagating errors," though related ideas had circulated in the applied mathematics and control theory literature even earlier. Backpropagation and gradient descent are often mentioned together so frequently that it's easy to conflate them, but they answer two different questions: backpropagation answers "what is the gradient, exactly, for every parameter in this network," and gradient descent answers "given that gradient, how should I update the parameters to reduce the loss."

That combination, backpropagation to compute the gradient, gradient descent (usually mini-batch, usually with an adaptive optimizer like Adam) to act on it, is the training mechanism underneath nearly every deep learning system currently in production: image classifiers, speech recognition systems, recommendation engines, and large language models alike. The scale has grown enormously, from Cauchy's small systems of astronomical equations in 1847 to models with hundreds of billions of trainable parameters today, but the underlying two-step loop, compute the local slope, take a small step against it, has not fundamentally changed.

How does gradient descent relate to model generalization?

Gradient descent only optimizes how well a model fits its training data; it says nothing directly about how well the model will perform on new, unseen data. A model can reach a very low training loss through gradient descent and still generalize poorly, a phenomenon covered in more depth in why overfitting hurts machine learning, which is why validation performance, not just training loss, has to be tracked separately during any real training run.

Common misconceptions about gradient descent

A persistent misunderstanding is that gradient descent always finds the single best possible answer, the global minimum, the absolute lowest point across the entire loss landscape, every time it's run. In practice, especially for the complex, non-convex loss surfaces typical of deep neural networks, gradient descent often converges to a good local minimum rather than a provably optimal global one. This sounds like a serious limitation until you consider the more recent theoretical and empirical finding, discussed above, that in very high-dimensional parameter spaces, most local minima found by well-tuned gradient descent tend to have loss values close to each other and close to the best achievable value, which is a large part of why deep learning works as reliably in practice as it does despite the underlying optimization problem being provably hard in the general case.

A second common misconception is that gradient descent needs to know the entire shape of the loss landscape in advance, the way a route-planning algorithm needs a full map before it can find the shortest path. It doesn't. The algorithm only ever needs the local slope at its current position, recalculated fresh at every single step, which is exactly why it works for functions so complex, high-dimensional, and non-convex that visualizing or solving them analytically is completely out of reach.

A third misconception, common among people newer to the field, is that a lower training loss always means a better model. Training loss measures fit to the data gradient descent is actually optimizing against, and a model can drive that number arbitrarily close to zero by memorizing training examples rather than learning patterns that generalize, which is exactly the failure mode that separate validation and test sets, discussed in why training and test data must be separated, are designed to catch.

Does a bigger model always need a smaller learning rate?

Not necessarily, and the relationship is more nuanced than a simple rule. Learning rate depends more on factors like batch size, the specific optimizer in use, and how the model's weights are initialized than on parameter count alone, which is why practitioners typically tune it empirically for a given architecture and dataset rather than deriving it purely from model size.

What ensemble methods do differently

It's worth briefly distinguishing gradient descent from a different but related family of techniques used elsewhere in machine learning. Not every model is trained by gradient descent at all: methods like random forests build many separate decision trees and combine their predictions through averaging or voting rather than iteratively minimizing a single differentiable loss function across shared parameters, an approach covered in how random forests reduce error. The two approaches solve the same underlying problem, getting a model to make accurate predictions, through fundamentally different mechanisms: one iteratively refines a shared set of parameters against a smooth loss surface, the other combines many independently trained, simpler models to cancel out each individual model's errors.

The takeaway

Gradient descent solves an optimization problem too large to tackle directly by never trying to see the whole landscape at once, it only ever asks "which direction is downhill from here," takes a step, and asks again. That simple, repeatable rule, first formalized by Cauchy for astronomy problems in 1847, scales up almost unchanged to training models with hundreds of billions of parameters today, with mini-batching, momentum, and adaptive optimizers like Adam layered on top to handle the noise, scale, and saddle-point-riddled terrain that modern deep learning throws at it.

---

Read next:

• Why overfitting hurts machine learning
• Why training and test data must be separated
• How random forests reduce error

FAQ

Q: Who invented gradient descent?

Augustin-Louis Cauchy is generally credited with the first written description of a steepest-descent method, presented to the French Academy of Sciences in 1847 in the context of solving systems of equations related to astronomical calculations. The technique was later adapted and formalized for statistical estimation, notably through Herbert Robbins and Sutton Monro's 1951 work on stochastic approximation methods.

Q: What is the difference between gradient descent and stochastic gradient descent?

Standard (batch) gradient descent calculates the exact gradient using the entire training dataset before each parameter update, which is accurate but computationally expensive on large datasets. Stochastic gradient descent estimates the gradient using one example, or in modern practice a small random mini-batch, at a time, trading some precision for much faster and more frequent updates.

Q: Why does the learning rate matter so much?

The learning rate controls how large a step the algorithm takes at each update. Too small a rate makes training impractically slow; too large a rate can cause the algorithm to overshoot the loss minimum, oscillate across the valley, or diverge to progressively worse performance instead of improving.

Q: Does gradient descent always find the best possible model?

Not necessarily. Complex, non-convex loss landscapes typical of deep neural networks contain local minima and saddle points, so gradient descent often converges to a good, but not provably globally optimal, solution. Research on high-dimensional loss landscapes suggests this is usually sufficient for strong practical performance, since most local minima found in practice tend to have similar, near-optimal loss values.

Q: How is gradient descent connected to backpropagation?

Backpropagation, popularized by Rumelhart, Hinton, and Williams in a 1986 Nature paper, is the algorithm that efficiently computes the gradient for every parameter in a multi-layer neural network by propagating error signals backward through the layers. Gradient descent then uses those computed gradients to actually update the parameters and reduce the loss.

Q: What does the Adam optimizer do differently from plain gradient descent?

Adam, introduced by Diederik Kingma and Jimmy Ba in 2014, combines momentum, a running average of past gradients, with adaptive, per-parameter learning rates based on estimates of the gradient's first and second moments. This lets it automatically adjust step sizes differently for each parameter, which typically speeds up convergence and reduces the amount of manual learning-rate tuning required compared to plain SGD.

Q: Why do practitioners use mini-batches instead of the full dataset or a single example?

Mini-batches balance two competing needs: batches large enough to give a reasonably stable, low-noise estimate of the true gradient, and batches small enough to fit in accelerator memory and allow frequent parameter updates. This makes mini-batch gradient descent the practical default for nearly all modern deep learning training.