How Random Forests Reduce Error

A single decision tree trained on medical diagnostic data can score near-perfect accuracy on the patients it trained on and then perform poorly on new patients, because it has memorized quirks specific to that training set rather than learning patterns that generalize. Statistician Leo Breiman, working at UC Berkeley, published a fix in 2001 that looks almost too simple to work: build hundreds of imperfect, randomized trees instead of one, and let them vote (Breiman, 2001, Machine Learning 45, 5–32). The method, called a random forest, is now one of the most widely used general-purpose algorithms in applied machine learning, and understanding why it works reveals something useful about error itself: it is often cheaper to cancel noise out than to eliminate it at the source.

Why one decision tree is a fragile decision-maker

A single decision tree is what statisticians call a high-variance model. Small changes to the training data, removing a handful of examples, adding a few new ones, can produce a substantially different tree structure and different final predictions, even though the underlying real-world relationship being modeled hasn't changed at all. This sensitivity is a direct consequence of how trees are built.

At every branch point, a decision tree algorithm looks at the data available at that node and picks the single split, a rule like "is patient age over 50?", that best separates the outcomes. This process is greedy, meaning it locks in the locally best choice at each step without reconsidering it later, and it is entirely determined by whatever data happened to land at that node. If one unusual data point tips an early split one way instead of another, every decision made further down that branch inherits the consequences of that one anomaly, and the tree's structure can diverge sharply from what a slightly different training sample would have produced.

This is why decision trees, left to grow deep and unconstrained, are notorious for overfitting. A deep tree can carve out a rule so specific it isolates a single training example, which is functionally the same as memorizing that example rather than learning a rule that transfers to new data. Pruning helps limit this, but pruning also throws away real signal along with the noise, and no single pruning depth is right for every dataset. The related SciHub101 explainer on how decision trees split data covers the information-gain mechanics behind each individual split in more depth; the point relevant here is that greedy, sequential splitting is precisely what makes a lone tree unstable.

Breiman's insight, formalized in the 2001 paper, was that instability itself could be turned into an advantage, but only if you had many differently unstable trees to average together rather than trying to build one stable tree from scratch.

Bootstrap aggregating: training each tree on a different slice of the same data

The first mechanism behind a random forest is bagging, short for bootstrap aggregating, a technique Breiman had introduced five years earlier in a separate 1996 paper (Breiman, 1996, Machine Learning 24, 123–140). Bagging works by drawing a random sample from the training set for each tree, sampled with replacement, meaning any given data point can be picked multiple times for one tree's sample while being left out entirely for another tree's sample. This resampled set is called a bootstrap sample, and it is typically the same size as the original training set even though it is built by repeated random draws rather than a straight copy.

Because each tree in the forest sees a slightly different version of the training data, each tree ends up making somewhat different mistakes. That property, differently distributed errors across trees, is the entire mathematical basis for why averaging helps. If every tree in the forest made exactly the same mistakes on exactly the same inputs, averaging their predictions would accomplish nothing: the errors would simply average back to themselves. It is only when errors vary from tree to tree, canceling rather than reinforcing one another, that the ensemble's combined prediction becomes more accurate than any individual tree's.

A useful way to picture this: imagine asking a hundred different physicians to each review a randomly incomplete subset of a patient's test results and history, then taking the majority verdict as the final diagnosis. Any single doctor working from a partial record might miss something important, and different doctors will miss different things depending on which parts of the record they happened to see. But a diagnosis that is genuinely well-supported by the full picture tends to show up consistently across most of the doctors' partial views, while any one doctor's blind spot gets outvoted by the others who didn't share it.

On average, about 63% of the original training examples appear at least once in any given bootstrap sample; the remaining roughly 37%, called "out-of-bag" data for that tree, can be used as a free internal validation set without setting aside separate holdout data (Breiman, 2001, *Machine Learning* 45, 5–32).

The second randomization: restricting which features each split can use

Bagging alone produces meaningfully better predictions than a single tree, but Breiman found it wasn't enough on its own to fully unlock the benefit of averaging, and the reason gets at something subtle about correlated errors. If one feature in the dataset is unusually predictive, most bagged trees will still independently discover that feature and split on it near the root of the tree, regardless of which bootstrap sample they were trained on. The result is a forest of trees that look more alike than random sampling alone would suggest, and trees that resemble each other also tend to make correlated mistakes, which limits how much averaging can help.

Random forests solve this with a second, distinct layer of randomness. At every single split, rather than considering every available feature and picking the single best one, the algorithm restricts the choice to a random subset of the features, typically the square root of the total feature count for classification tasks, and forces the tree to choose its best split from within that limited subset. Because the subset is redrawn randomly at every split, different trees, and even different branches within the same tree, end up relying on different, often less individually powerful features to make their decisions.

Randomization step	What it varies	What it decorrelates
Bootstrap sampling (bagging)	Which training rows each tree sees	Training data across trees
Random feature subsampling	Which columns each split can consider	Split choices within and across trees
Majority vote / averaging	Final prediction combination method	Individual trees' output noise

Two separate randomization steps work together: bagging varies the data each tree sees, feature subsampling varies the logic each split can use, and the combination is what pushes tree errors apart from each other rather than letting them cluster around the same few dominant patterns.

This feature-subsampling step is the specific innovation that separates a random forest from plain bagged trees, and it is the part of the 2001 paper that gave the algorithm its name. Bagging alone randomizes what data a tree sees; feature subsampling randomizes what logic a tree is even allowed to consider at each decision point. The combination deliberately decorrelates trees from one another in a way that resampling the data alone cannot achieve, because it prevents dominant features from asserting the same influence across the entire forest.

Why decorrelation matters more than raw tree count

Averaging N independent, equally accurate estimators reduces error variance roughly in proportion to how uncorrelated those estimators' errors are, not simply how many of them you have. A forest of 500 highly similar trees that mostly agree on the same reasoning provides far less error reduction than a forest of 500 genuinely diverse trees, even though both forests have identical tree counts. This is why the feature-subsampling parameter, not just the number of trees, is one of the most consequential hyperparameters to tune in a random forest.

What the forest actually predicts

The averaging mechanism differs slightly depending on the type of task. For classification problems, where the model predicts a category such as "spam" or "not spam," each tree in the forest casts one vote for the class it predicts, and the forest's final output is whichever class receives the most votes across all trees. Some implementations instead average each tree's predicted class probabilities and take the highest-probability class, which tends to produce smoother, better-calibrated confidence scores than a simple majority vote.

For regression problems, where the model predicts a continuous number, such as a house price or a patient's expected recovery time, there is no voting involved. Instead, the forest takes the plain numeric average of every individual tree's predicted value. A tree that badly overestimates and a tree that badly underestimates, if both are represented in the forest, partially cancel each other out in the final average, which is exactly the noise-cancellation effect Breiman's method is designed to produce.

Either way, the underlying logic is identical: many individually noisy, imperfect estimates are combined into a single estimate that is, on average, closer to the true value than any individual tree's guess would have been. This is also why random forests tend to be relatively robust to individual outlier trees. A handful of trees trained on unlucky bootstrap samples that happen to produce a badly overfit structure get outweighed by the majority of trees that were not thrown off by that particular anomaly.

Why more trees help, but only up to a point

Adding more trees to a random forest generally improves the stability of its predictions, because averaging over a larger number of roughly independent estimates smooths out more of the random noise contributed by any single tree. In practical terms, this means predictions from a forest of 500 trees will typically vary less from one training run to the next than predictions from a forest of only 10 trees, even when trained on the exact same dataset with the exact same algorithm settings.

That improvement, however, has diminishing returns. Once a forest reaches a few hundred trees, adding still more trees usually produces only marginal gains in accuracy while continuing to increase both training time and memory footprint, since every additional tree has to be stored and queried at prediction time. Practitioners typically tune the number of trees against a validation set and stop increasing it once accuracy gains flatten out, rather than assuming that more is unconditionally better.

One genuinely useful property here, and a point Breiman emphasized in the original paper, is that random forests are notably resistant to overfitting as tree count grows, in contrast to how adding complexity to a single decision tree, more depth, more splits, tends to make overfitting worse. Averaging additional trees together does not add complexity to how the model handles any individual prediction the way adding more splits deepens a single tree's decision path. This is one reason random forests remain a reasonable default choice even for practitioners who aren't deeply experienced in hyperparameter tuning; the SciHub101 explainer on why overfitting hurts machine learning covers what overfitting costs in models that lack this kind of built-in resistance.

How this compares to a single, carefully tuned tree

A single, carefully pruned decision tree can occasionally match a random forest's accuracy on simple, low-noise datasets, but it requires more careful manual tuning to avoid both overfitting (too deep) and underfitting (too shallow), and it remains vulnerable to instability from small data changes in a way a forest of hundreds of trees is not. The forest trades some of that manual tuning burden for a heavier computational cost at both training and prediction time.

What you trade away for that added stability

Random forests are meaningfully harder to interpret than a single decision tree, and this tradeoff matters in regulated or high-stakes domains where practitioners need to explain why a model made a specific prediction. A lone decision tree can be drawn out and read almost like a flowchart: follow the branches from root to leaf, and the exact sequence of conditions behind any single prediction is fully visible.

A forest built from several hundred trees voting or averaging together has no equivalent single readable path. There is no one sequence of if-then conditions to point to, because the final prediction is the aggregate outcome of hundreds of separately reasoned trees, many of which relied on different features entirely due to the random feature-subsampling step.

Because of this, practitioners typically fall back on a separate diagnostic tool called feature importance, a score computed by measuring how much each feature's splits, averaged across the entire forest, contributed to reducing prediction error. Feature importance scores can tell you that "income" mattered more than "zip code" across the forest as a whole, but they cannot tell you the specific reasoning chain behind any one individual prediction the way a single tree's decision path can. This gap between aggregate explainability and per-prediction explainability is a genuine limitation, not a minor inconvenience, and it is one reason simpler, more interpretable models are sometimes preferred even when they are somewhat less accurate.

Computational cost is the other real tradeoff. Training and querying hundreds of trees requires substantially more memory and compute than training and querying one tree, and while random forests parallelize well, since each tree can be trained independently of the others, the resource cost is still nontrivial for very large datasets or latency-sensitive applications.

Common misconceptions about random forests

A common misunderstanding is that random forests simply run the same decision tree algorithm many times over. In reality, each tree in the forest is deliberately made different from the others through two distinct randomization mechanisms working together: bootstrap sampling varies the training data each tree sees, and feature subsampling varies which features each split within that tree is even allowed to consider. Both mechanisms exist specifically to push the trees' individual errors apart from one another, since correlated errors would defeat the entire purpose of averaging.

Another misconception is that random forests eliminate overfitting entirely. They substantially reduce the overfitting risk that plagues a single unconstrained decision tree, and Breiman's original paper demonstrated this empirically across multiple benchmark datasets, but "substantially reduce" is not the same as "eliminate." Poorly chosen hyperparameters, such as allowing individual trees to grow to unlimited depth, letting each split consider too large a fraction of the total features, or training with far too few trees, can still leave meaningful overfitting risk inside a random forest, particularly on smaller datasets where bootstrap samples overlap heavily with each other by chance.

A third misconception worth addressing is that random forests always outperform simpler models. On datasets with strong linear structure, or on very small datasets where there isn't enough data to build genuinely diverse bootstrap samples, simpler models such as linear or logistic regression can match or beat a random forest while remaining far easier to interpret and far cheaper to train. Random forests earn their reputation on messy, nonlinear, feature-rich data where the relationships between inputs and outputs are hard to specify by hand in advance, not as a universal default for every problem.

The takeaway

Breiman's contribution was not a new kind of tree; it was the recognition that averaging many differently flawed models cancels out more error than any single effort to build one perfect model ever could. Bootstrap sampling and random feature selection are the two specific mechanisms that force individual trees to be different enough from one another for that averaging to actually pay off, rather than simply averaging the same mistake five hundred times over. The result is a method that trades some interpretability and some computational efficiency for a large gain in stability, turning a notoriously fragile, high-variance predictor into one of the most reliably strong general-purpose algorithms available in applied machine learning today.

---

Read next:

• How decision trees split data
• Why overfitting hurts machine learning
• What model drift means in production

FAQ

Q: Who invented random forests?

Leo Breiman, a statistician at UC Berkeley, formalized the random forest algorithm in a 2001 paper published in the journal Machine Learning (Breiman, 2001, Machine Learning 45, 5–32), building on his earlier 1996 paper introducing bootstrap aggregating, or bagging (Breiman, 1996, Machine Learning 24, 123–140).

Q: Why does averaging many trees reduce error compared to using one tree?

A single decision tree is highly sensitive to small changes in training data, which produces inconsistent predictions from one training run to the next. Averaging predictions across many trees, each trained on a different bootstrap sample and restricted to different random subsets of features at each split, cancels out much of each individual tree's noise while patterns supported across most of the trees survive.

Q: What is the difference between bagging and a random forest?

Bagging trains multiple trees on bootstrapped, randomly resampled versions of the training data and then averages or votes on their predictions. Random forests add a second, independent layer of randomness on top of bagging: at every split, each tree is restricted to considering only a random subset of the available features, which further decorrelates the trees' errors and produces a larger accuracy gain than bagging alone.

Q: Do random forests eventually stop improving as you add more trees?

Yes. Prediction stability keeps improving as more trees are added to the forest, but the rate of improvement diminishes quickly, and beyond roughly a few hundred trees, additional gains are typically marginal while training time and memory usage keep rising. Most practitioners tune the tree count against a validation set rather than maximizing it indefinitely.

Q: Why are random forests harder to interpret than a single decision tree?

A single tree's decision path can be drawn out and followed step by step like a flowchart. A random forest's prediction instead comes from combining the votes or averaged outputs of hundreds of separately trained trees, many of which relied on different features due to random feature subsampling, so there is no single readable path behind any one prediction. Practitioners typically rely on aggregate feature-importance scores instead, which explain overall patterns across the forest but not the specific reasoning behind an individual prediction.

Q: Can random forests overfit?

Yes, though far less easily than a single unconstrained decision tree. Overfitting risk in a random forest typically comes from letting individual trees grow to unlimited depth, allowing each split to consider too large a share of the available features, or training with too few trees, particularly on small datasets where bootstrap samples inevitably overlap heavily with one another.

Q: Are random forests always better than a single decision tree?

Not necessarily. Random forests generally produce more stable, accurate predictions on messy, nonlinear datasets, but they sacrifice the direct interpretability of a single tree and require more compute to train and query. On very small datasets or problems with strong linear structure, a single well-tuned tree or an even simpler linear model can sometimes match a random forest's accuracy while remaining far easier to explain.