PCA Explained: How Dimensionality Reduction Works

Principal component analysis (PCA) takes a dataset with many measurements per item, finds the few directions along which the items differ most, and re-describes every item using only those directions. A table with 1,000 columns can become a table with 2, 10, or 50, while keeping as much of the original spread as possible. That is why PCA shows up in genomics, image compression, finance, and nearly every tutorial that draws a scatter plot of high-dimensional data. This article explains what PCA computes, how to choose how many components to keep, and where it, along with its fashionable cousins t-SNE and UMAP, quietly misleads.

The curse of dimensionality, in numbers

Dimensionality reduction exists because adding columns makes a dataset behave badly, and the problem gets worse faster than intuition suggests.

Start with space. Suppose your data fills a unit cube uniformly, and you want a smaller cube around a point that captures 10% of the data. In one dimension, you need an interval 0.1 wide. In 10 dimensions, the cube's edge must be 0.1 raised to the power 1/10, which is about 0.79 of the full range. In 100 dimensions it is about 0.98. A "local neighborhood" that holds a tenth of the points now stretches across nearly the whole range of every feature, so the word "local" has stopped meaning anything. (These figures come from straightforward arithmetic on a uniform cube, not from a particular dataset.)

Now consider coverage. Cut each axis into 10 bins. One feature gives you 10 cells to fill with data. Two features give 100. Ten features give 10 billion. Real datasets almost never have enough rows to populate a grid like that, so most of the space is empty, and models that rely on similarity between points (nearest neighbors, clustering, kernel methods) find that nearly everything is far from everything else. A related effect is that models with many inputs have more room to memorize noise, which is the failure described in overfitting and underfitting.

There is a second, more hopeful fact. Real data rarely uses all its dimensions. Pixel values in a photo of a face are heavily correlated with their neighbors. Gene expression levels move together because genes work in pathways. Prices of related assets rise and fall together. The data may be listed in 1,000 columns but lie close to a much smaller structure, like a sheet of paper crumpled in a room: three-dimensional space, but a two-dimensional object. PCA is a method for finding the flat sheet when it is roughly flat.

Variance as the thing to keep

PCA needs a definition of "information worth keeping," and it picks variance: the spread of the data along a direction. The idea is that if a measurement barely differs between items, it cannot help tell them apart, while a direction along which items spread widely is probably carrying signal.

An analogy: you are photographing a long, thin sculpture, and you can take only one picture. You would not shoot it end-on, where it looks like a small circle. You would walk to the side where the full length shows. PCA does exactly that for a cloud of data points. It looks for the viewing angle from which the cloud appears most spread out, takes that as the first axis, and then looks for the next-best angle at a right angle to the first, and so on.

Two properties define the result:

1. The components are orthogonal. Each new axis is perpendicular to the previous ones, so the new coordinates are uncorrelated with one another.
2. They are ordered by variance. The first principal component captures the most variance of any single direction, the second captures the most of what is left, and so on.

Karl Pearson described the geometric version of the idea in 1901 as finding lines and planes of closest fit to points in space, and Harold Hotelling formalized it as principal components in 1933. Ian Jolliffe and Jorge Cadima's 2016 review in Philosophical Transactions of the Royal Society A frames the modern view: PCA builds new uncorrelated variables that successively maximize variance, and finding them reduces to an eigenvalue problem.

Two views of the same answer

Maximizing variance and minimizing reconstruction error turn out to be the same goal. Pearson's picture was "the line closest to all the points," measured by the sum of squared perpendicular distances. Hotelling's was "the direction with the most spread." The Pythagorean theorem links them: for each point, the squared distance from the center equals its squared length along the line plus its squared distance from the line. The total is fixed, so a line that captures more spread necessarily leaves less distance behind. Choosing one automatically chooses the other.

Eigenvectors and the covariance matrix, intuitively

The computation is where PCA acquires its reputation for scary linear algebra, but the mechanics are short.

Step 1: center the data. Subtract each column's mean so the cloud is centered at the origin. Scikit-learn's documentation states that its PCA "centers but does not scale the input data for each feature before applying the SVD." The scaling part will matter in a moment.

Step 2: measure how features vary together. The covariance matrix is a square table with one row and column per feature. The diagonal holds each feature's variance, and the off-diagonal entries hold how pairs of features move together. A positive entry means they rise and fall together.

Step 3: find the matrix's eigenvectors. An eigenvector of the covariance matrix is a direction that the matrix only stretches, never rotates. The amount of stretch is the eigenvalue. For a covariance matrix, the eigenvector with the largest eigenvalue points along the direction of greatest variance, and the eigenvalue equals the variance along it. Those eigenvectors are the principal components.

In practice, libraries skip forming the covariance matrix and apply a singular value decomposition (SVD) to the centered data directly, which is numerically steadier and gives the same components. Scikit-learn's PCA works this way.

A two-feature example you can check by hand

Take two features with variances 4 and 1 and a covariance of 1.5 between them. The covariance matrix is:

[[4.0, 1.5],
 [1.5, 1.0]]

The trace (4 + 1 = 5) equals the sum of the eigenvalues, and the determinant (4 times 1 minus 1.5 squared = 1.75) equals their product. Solving gives eigenvalues of about 4.62 and 0.38. The first component therefore carries 4.62 out of a total variance of 5, or about 92.4%, and the second carries the remaining 7.6%.

The first eigenvector points along a line where y is about 0.414 times x, which is an angle of 22.5 degrees from the first feature's axis. In plain terms, PCA noticed that the two features are positively related, tilted the axes to lie along that relationship, and found that almost everything interesting happens along one of the new axes. Dropping the second axis turns a 2D scatter into a 1D number line and discards about 7.6% of the spread.

What the output looks like in code

Here is the standard workflow in scikit-learn, using the iris dataset (150 flowers, four measurements each):

from sklearn.datasets import load_iris
from sklearn.decomposition import PCA

X = load_iris().data
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)

print(pca.explained_variance_ratio_)  # variance share per component
print(pca.components_)                # the axes, as weights on original features

Scikit-learn's own LDA-versus-PCA example fits exactly this on the raw iris features and prints an explained variance ratio of about [0.9246, 0.0531] for the first two components. That is 92.5% and 5.3%, or roughly 97.8% combined. A four-column table has been replaced by two columns that preserve almost all of its spread.

Notice what components_ gives you: each new axis written as a weighted mix of the original features. That table is how you interpret a component. If the first component puts large positive weights on petal length, petal width, and sepal length, you can call it roughly "overall flower size." The scikit-learn gallery's iris example makes a related point: petal width and length are the most discriminating features, and the species can be told apart using only the first principal component.

Be careful with that interpretation, though. A component is a mathematical optimum, not a promise of meaning. Sometimes the first component lines up with something nameable such as size; often it does not.

Choosing how many components to keep

PCA gives you as many components as original features. The reduction comes from keeping only the first k. There are three common ways to choose k, and none of them is the single correct answer.

Method	How it works	Best for	Watch out for
Cumulative variance threshold	Keep enough components to reach, say, 90% or 95% of total variance	Compression, preprocessing for a model	The threshold is a convention, not a law
Scree plot elbow	Plot each component's variance in order; keep components before the curve flattens	Exploratory analysis	Elbows are often ambiguous
Downstream performance	Try several k values and cross-validate the model that uses them	Supervised pipelines	Costs more compute

The scree plot takes its name from the pile of loose rock at the base of a cliff: the steep part is the cliff (the components that matter), and the flat tail is rubble. When the plot shows a clear break, the choice is easy. When the curve declines gradually, which is common in noisy real data, you are choosing a trade-off between simplicity and fidelity, and you should say so.

For prediction tasks, the third method is usually the most defensible, because the goal is a good model, not a pretty variance plot. This is where a discipline from feature engineering applies directly: PCA is a feature-construction step, and its settings should be judged by what they do to model performance on held-out data.

Keep the PCA inside the training fold

One mistake is common enough to deserve its own warning. If you run PCA on the entire dataset and then split into training and test sets, the components were computed using information from the test rows. The test data influenced the transformation, and your evaluation is mildly optimistic. This is a form of the problem in why data leakage breaks AI evaluations. The fix is to fit PCA on the training data only, then apply the learned transformation to the test data, which is easiest to guarantee by placing PCA inside a scikit-learn Pipeline.

Worked example: eigenfaces

The most famous application of PCA to real images is eigenfaces, from Matthew Turk and Alex Pentland's 1991 paper "Eigenfaces for Recognition" in the Journal of Cognitive Neuroscience.

A grayscale photo that is 100 by 100 pixels has 10,000 values, so it is one point in a 10,000-dimensional space. (The image size is an illustration; the paper's own images differ.) Treat each photo of a face as such a point, collect many aligned face photos, and run PCA. The principal components are themselves 10,000-value vectors, which means you can display them as images. Turk and Pentland called them eigenfaces because they are the eigenvectors of the set of faces. As they note, the eigenfaces "do not necessarily correspond to features such as eyes, ears, and noses."

Any face can then be described as a weighted sum of eigenfaces, so a face becomes a short list of weights instead of 10,000 pixel values. To recognize someone, the system compares that short weight vector with the weights of known individuals. The paper's system relied on upright, frontal views and treated recognition as an intrinsically two-dimensional problem.

The method has obvious limits: it is sensitive to lighting, pose, and alignment, and modern face recognition uses deep networks. Eigenfaces survive in textbooks because they make the machinery visible. The components are images; the weights are coordinates; compression and recognition both fall out of the same decomposition. The same logic underlies later work on compressed representations such as the embeddings used for similarity search, though embeddings are learned by neural networks and are not simply linear projections.

When PCA fails

PCA has an excellent record, and a short list of failure modes that everyone who uses it should know.

Unscaled features dominate. Because PCA maximizes variance and variance depends on units, a feature measured in large numbers swamps one measured in small numbers. Put household income in dollars next to age in years and the first component will be almost entirely income, because its raw variance is millions of times larger. This is a statement about units, not about importance. The standard remedy is to standardize each feature (subtract the mean, divide by the standard deviation) before PCA, so each starts with variance 1. Scikit-learn's PCA does not do this for you. The iris result above used raw features that share the same unit (centimeters), which is why it was reasonable there; with mixed units, it would not be.

Structure that is curved, not flat. PCA only finds linear structure. If the data lies along a spiral, a ring, or the famous "Swiss roll," no straight axis captures it, and two components can look like a mess even though the data is really one-dimensional. Scikit-learn documents Kernel PCA as an extension that achieves nonlinear dimensionality reduction through kernels, and manifold-learning methods go further.

Variance is not relevance. The direction with the most spread is not necessarily the direction that separates your classes. A feature that matters for the prediction but varies little can be discarded because PCA never looked at the labels. In scikit-learn's LDA-versus-PCA comparison, the difference is described in one line: PCA is unsupervised, while linear discriminant analysis uses class labels to find directions that account for variance between classes.

Outliers pull the axes. Variance is calculated with squared deviations, so a few extreme points can drag a component toward themselves. Check for outliers, or consider robust variants, before trusting the axes.

Interpretability is lost. After PCA, columns like "component 3" replace columns like "blood pressure." You can inspect the weights, but a model built on components is harder to explain to a stakeholder than one built on named features. If interpretability is a requirement, the loss may outweigh the compression.

Related to the clustering use case: running PCA before clustering is a common pipeline, since distance-based methods suffer in high dimensions, and it removes some noise. But the compression changes distances, so the clusters you find are clusters in the reduced space. See how clustering finds hidden groups for the downstream step, and ML algorithms explained for where PCA sits among supervised and unsupervised methods.

Scaling PCA to large data

For a table that fits in memory, exact PCA is fine. Larger problems have options built into scikit-learn.

• Randomized PCA (svd_solver='randomized') approximates only the top singular vectors. The documentation gives its complexity as O(n_max² · n_components), against O(n_max² · n_min) for the exact method, which helps a great deal when you want 20 components out of thousands of features. It also warns that inverse_transform with this solver "is not the exact inverse transform of transform."
• Incremental PCA processes data in minibatches. Plain PCA supports only batch processing, so all data must fit in memory; IncrementalPCA "allows for partial computations which almost exactly match the results of PCA," and its memory use depends on batch size, not on total dataset size.
• Whitening (whiten=True) rescales each component to unit variance, which the documentation notes is useful for downstream models that assume isotropic data, such as RBF-kernel SVMs and K-Means.

t-SNE and UMAP: what the pretty plots do and do not mean

PCA is often replaced, for visualization, by t-SNE (introduced by Laurens van der Maaten and Geoffrey Hinton in 2008) or UMAP (Leland McInnes, John Healy, and James Melville, first posted February 9, 2018). Both are nonlinear: they try to place points in 2D so that points that were neighbors in high dimensions stay neighbors. The pictures are attractive, with islands of color, and that is exactly the danger.

The UMAP authors' abstract claims it is competitive with t-SNE in visualization quality, arguably preserves more global structure, and has superior run time. Those are the authors' own claims, and you should read them as such. Meanwhile, Martin Wattenberg, Fernanda Viégas, and Ian Johnson's Distill article "How to Use t-SNE Effectively" (October 13, 2016) lists several ways t-SNE plots mislead:

• Hyperparameters matter. Perplexity, a setting that roughly controls the neighborhood size, changes the picture substantially, and the authors stress iterating until the plot reaches a stable configuration.
• Cluster sizes mean nothing. t-SNE equalizes density, so a widely spread cluster can appear as small as a tight one.
• Distances between clusters may mean nothing. The authors write that distances between well-separated clusters in a t-SNE plot may carry no information.
• Random noise can look structured. At low perplexity, pure Gaussian noise can produce convincing clumps.
• One plot is not enough. Some structure shows only at certain perplexities, so the authors recommend looking at several.

The practical rule: use t-SNE or UMAP to generate hypotheses ("these samples seem to form two groups"), and then test the hypothesis with something that does not depend on the picture. Do not measure cluster sizes or between-cluster gaps from the plot. PCA is less flashy, but its axes have a definition you can state in one sentence, and distances in a PCA plot are real distances between projections, which makes it easier to reason about.

A common pairing combines them: run PCA first to reduce thousands of features to 30 or 50, which removes noise and speeds up the nonlinear method, then apply t-SNE or UMAP to those components for the final 2D picture. That is a convention rather than a theorem, and it brings back the question of how many components to keep.

Common misconceptions

"PCA selects the most important features." It does not select original features at all. It builds new ones, each a mix of all the originals. If you need to pick a subset of original columns, that is feature selection, a different tool.

"The components with the most variance predict the target best." Not necessarily, as discussed above. PCA never sees the labels.

"PCA removes noise." It can, when noise has low variance and signal has high variance, which is the assumption behind keeping top components. If the noise is large or the signal is subtle, the discarded components may contain the signal.

"A 2D PCA plot shows everything." It shows the projection onto two axes. If those two components explain only 30% of the variance, the picture hides most of the data, and points that look close may be far apart in the dropped dimensions. Always report the explained variance next to the plot.

"PCA works on any data." It assumes numeric features and linear relationships, and it is sensitive to scale. Categorical data needs encoding first, and the encoding choices change the result.

Frequently asked questions

Q: What does principal component analysis do in simple terms? A: PCA finds new axes for your data, ordered by how much of the data's variance each captures, and lets you keep only the first few. The result is a smaller table that preserves as much of the original spread as possible, using fewer columns.

Q: How many principal components should I keep? A: There is no universal number. Common approaches are keeping enough components to reach a cumulative variance threshold such as 90% or 95%, choosing the point where a scree plot flattens, or cross-validating a downstream model across different values. For prediction, the cross-validated choice is usually the most defensible.

Q: Do I need to scale my data before PCA? A: Usually yes, when features have different units. PCA maximizes variance, so a feature with large numeric values dominates. Scikit-learn's PCA centers the data but does not scale it, so you standardize features yourself before fitting.

Q: What is the difference between PCA and t-SNE? A: PCA is a linear method whose axes are weighted mixes of the original features and whose distances are meaningful projections. t-SNE is nonlinear and tries to keep neighbors together in 2D, but its cluster sizes and between-cluster distances can be misleading, and its output depends strongly on settings such as perplexity.

Q: Is PCA supervised or unsupervised? A: Unsupervised. It uses only the features, never the labels. Linear discriminant analysis is the supervised counterpart that uses class labels to find directions separating classes.

Q: Can PCA be used on images? A: Yes. Each image becomes one long vector of pixel values, and PCA finds components that are themselves image-shaped. Turk and Pentland's 1991 eigenfaces paper applied this to face recognition, representing each face as a weighted sum of eigenfaces.

Q: Why not just drop the low-variance features instead of using PCA? A: Dropping columns removes whole original features and ignores correlations between them. PCA combines correlated features into shared components, so it can compress groups of related measurements into a few axes without discarding any one measurement outright.

The takeaway

PCA works because real data is usually redundant. When many columns move together, a few well-chosen axes can describe the spread, and linear algebra finds those axes exactly: they are the eigenvectors of the covariance matrix, ranked by variance. That is why it can shrink iris to two columns while keeping about 97.8% of the variance, and why a face can be written as a short list of weights.

The same definition explains the failures. PCA cares about variance, so it is hostage to units, indifferent to labels, and blind to curved structure. Treat the number of components as a modeling decision tested on held-out data, fit the transformation on training data only, and read t-SNE and UMAP pictures as suggestions rather than measurements. For the surrounding toolkit, the complete machine learning guide shows where dimensionality reduction fits in a full workflow.

Sources

• scikit-learn User Guide, "Decomposing signals in components (matrix factorization problems)," PCA section: https://scikit-learn.org/stable/modules/decomposition.html
• scikit-learn, "Comparison of LDA and PCA 2D projection of Iris dataset": https://scikit-learn.org/stable/auto_examples/decomposition/plot_pca_vs_lda.html
• scikit-learn, "Principal Component Analysis (PCA) on Iris Dataset": https://scikit-learn.org/stable/auto_examples/decomposition/plot_pca_iris.html
• Wattenberg, Viégas, and Johnson, "How to Use t-SNE Effectively," Distill, October 13, 2016: https://distill.pub/2016/misread-tsne/
• McInnes, Healy, and Melville, "UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction," arXiv:1802.03426: https://arxiv.org/abs/1802.03426
• Jolliffe and Cadima, "Principal component analysis: a review and recent developments," Philosophical Transactions of the Royal Society A, 2016 (DOI 10.1098/rsta.2015.0202): https://royalsocietypublishing.org/doi/10.1098/rsta.2015.0202
• Turk and Pentland, "Eigenfaces for Recognition," Journal of Cognitive Neuroscience 3(1), 1991 (DOI 10.1162/jocn.1991.3.1.71): https://doi.org/10.1162/jocn.1991.3.1.71