How Image Recognition Works: The Science of Convolutional Neural Networks

In the first year of the ImageNet challenge, 2010, the best system failed to include the correct label among its five top guesses on roughly 28 percent of photographs. Two years later a convolutional neural network (CNN) called AlexNet cut that to 15.3 percent. By 2015, ResNet reached 3.57 percent, and in 2017 the winning entry hit 2.251 percent. Most of that progress traces to one architectural idea: instead of wiring every pixel to every neuron, let a small pattern detector slide across the whole image and reuse the same weights everywhere. This article explains how that works, step by step.

Key takeaways

• Convolutions detect local patterns. A small grid of learned numbers, called a filter, slides across the image and reports where its pattern appears.
• Weight sharing keeps the network small. One 3×3 filter on a color image has 27 weights no matter how large the picture is.
• Layers build a hierarchy. Early layers respond to edges and color, middle layers to textures and parts, and deep layers to whole objects.
• Depth required a trick. Very deep plain networks became harder to train, and ResNet's skip connections fixed that in 2015.
• Accuracy is not understanding. CNNs trained on ImageNet lean heavily on texture, and tiny pixel changes can flip their answers.
• CNNs are not obsolete. Vision Transformers match them at large scale, but modernized CNNs such as ConvNeXt still compete.

Why feeding raw pixels to an ordinary network fails

A 224×224 color photo contains 224 × 224 × 3 = 150,528 numbers, one for each color channel of each pixel. A standard fully connected network connects every input to every neuron in the next layer. With only 1,000 neurons in the first hidden layer, that is about 150 million weights before the network has learned anything useful.

The cost is only half the problem. A fully connected layer has no idea that two neighboring pixels are related. Shift a handwritten "7" three pixels to the right and every input changes, so the network must relearn the digit at every position. It is like teaching someone to recognize a friend by memorizing a separate description for every spot in a room where the friend might stand.

Images have structure that a good architecture can exploit. Nearby pixels are strongly correlated, because edges and textures span several pixels. And the same pattern, a curve or a corner, can appear anywhere in a picture and should be detected the same way wherever it appears. CNNs build those two facts into the design.

Convolution: sliding a pattern detector across the image

A convolutional layer applies a small matrix, called a filter or kernel, at every position of the image. At each position it multiplies the filter's numbers by the pixel values underneath, adds the products, and writes one output number. (Strictly speaking, deep learning libraries compute cross-correlation, which is convolution without flipping the filter, but the name stuck.)

Here is a filter that detects horizontal edges:

-1 -1 -1
 0  0  0
 1  1  1

Suppose it lands on a patch where the top row of pixels is dark (value 10) and the bottom row is bright (value 200), with the middle row also dark. The calculation is -(10 + 10 + 10) + 0 + (200 + 200 + 200) = 570, a strong response. On a flat patch where all nine pixels are equal, the positives and negatives cancel and the output is 0. The filter is silent on smooth regions and loud on a dark-above-bright boundary.

Slide the filter across the whole image and record every output, and you get a feature map: a grid showing where in the picture that pattern appeared. A layer usually applies dozens or hundreds of filters at once, so it produces a stack of feature maps, one per filter. A layer with 64 filters outputs 64 maps, each tracking a different pattern.

Weight sharing and the parameter savings

The same filter is applied at every position, so its numbers are reused everywhere. That is weight sharing, and it is the source of the savings. A 3×3 filter over a three-channel color image has 3 × 3 × 3 = 27 weights plus one bias term. A layer of 64 such filters has 64 × 28 = 1,792 parameters, regardless of whether the image is 32 pixels wide or 2,000.

Compare that with the fully connected layer above, which needed roughly 150 million weights for the same input. The Wikipedia entry on convolutional neural networks gives a smaller version of the same arithmetic: a fully connected neuron looking at a 100×100 image needs 10,000 weights, while a convolutional unit with a 5×5 shared window needs 25.

Stride, padding, and output size

Two settings control how the filter moves. The stride is the step size: a stride of 1 moves the filter one pixel at a time, a stride of 2 skips every other position and halves the output's width and height. Padding adds a border of zeros around the image so the filter can be centered on edge pixels and the output keeps the input's size.

For an input of width W, a filter of size K, padding P, and stride S, the output width is (W − K + 2P) / S + 1. With a 224-pixel input, a 3×3 filter, padding of 1, and stride 1, the output is (224 − 3 + 2) / 1 + 1 = 224. The size is preserved. With stride 2, it becomes 112.

Equivariance versus invariance

CNN explainers often say convolution gives "translation invariance." More precisely, convolution gives translation equivariance: if the object moves in the image, its pattern of activations moves by the same amount in the feature map. True invariance, giving the same final answer wherever the object sits, comes from later steps such as pooling and the final classifier. The difference matters because it explains why CNNs handle small shifts well but still fail on large changes in scale or viewpoint.

From edges to objects: how layers build a hierarchy

A single layer can only detect simple patterns. The power comes from stacking. The first convolutional layer looks at raw pixels and learns filters for oriented edges, color gradients, and small blobs. The second layer looks at the first layer's feature maps, not at pixels, so it can combine edges into corners, curves, and simple textures.

By the middle layers the filters respond to things like fur, mesh, or a circular part. In the deepest layers, units respond to whole object parts or categories: a dog's face, a wheel, a keyboard. Each layer sees a larger patch of the original image than the one before, a growing receptive field.

Stacking small filters is also cheaper than using one large one. Two stacked 3×3 layers cover the same 5×5 region as one 5×5 filter, and three stacked 3×3 layers cover 7×7. With C channels throughout, three 3×3 layers use 3 × 9 × C² = 27C² weights, while a single 7×7 layer uses 49C². The stack uses fewer parameters and adds extra nonlinearities between the layers, which lets it represent more complex functions. That reasoning is behind the design of the VGG networks discussed below.

Seeing what the filters learned

Researchers have checked the hierarchy directly. In 2013, Matthew Zeiler and Rob Fergus introduced a visualization technique that shows what stimulus activates intermediate layers of an ImageNet-trained network, and used it to diagnose and improve the architecture, in their paper Visualizing and Understanding Convolutional Networks. The pictures bore out the layered picture: simple structure early, complex structure late.

The design also echoes biology. Hubel and Wiesel's 1960s recordings from the visual cortex found simple cells that respond to oriented edges and complex cells that pool over them, which inspired early convolutional architectures; the story is told in the article on the Hubel and Wiesel visual cortex experiments. The comparison is an inspiration, not an equivalence. A CNN is a numerical function tuned by gradient descent, and the brain's wiring is far messier.

Pooling and the rest of the building blocks

A modern CNN is built from a handful of repeating parts.

Component	What it does	Why it is there
Convolution	Slides learned filters over the input	Detects local patterns with shared weights
Nonlinearity (usually ReLU)	Sets negative values to zero	Lets stacked layers compute more than a single linear map
Pooling	Summarizes small windows, such as 2×2, with the maximum or average	Shrinks the maps and tolerates small shifts
Normalization	Rescales activations in each batch	Stabilizes and speeds training
Fully connected or global-average layer	Combines the final features into class scores	Produces the answer
Softmax	Turns scores into probabilities that sum to 1	Gives a readable output such as 87% "tabby cat"

Max pooling keeps the largest value in each window and discards the rest. If a feature fires a pixel or two off its usual location, the maximum in the window is unchanged, which gives some tolerance to small shifts, and the maps shrink by a factor of four per 2×2 pooling step, which cuts computation. Max pooling was introduced in 1990, according to the Wikipedia history of CNNs. Global average pooling goes further, collapsing each feature map to one number before the final layer, a trick used in ResNet to avoid huge fully connected layers.

How a CNN learns

None of the filter values are written by hand. The network starts with random weights and sees labeled images. For each image it produces probabilities, a loss function measures how far they are from the right label, and backpropagation computes how much each weight contributed to the error. An optimizer such as stochastic gradient descent nudges every weight slightly in the direction that lowers the loss. The mechanics of that update are covered in how gradient descent finds better answers.

After millions of such nudges across a large dataset, the filters settle into useful detectors. The filter values in the first layer typically look like oriented edges and color blobs, and nobody told the network to learn them. This is the central change from older computer vision, which relied on human-designed features. The wider idea is explained in what machine learning is, and the general structure of layered networks in how neural networks work.

A short history: from the neocognitron to AlexNet

The ideas behind CNNs are older than most people assume. According to the Wikipedia history of CNNs, Kunihiko Fukushima's neocognitron in 1980 introduced alternating convolutional and downsampling layers, Wei Zhang and colleagues trained a shift-invariant character-recognition network with backpropagation in 1988, and Yann LeCun's group applied backpropagation to learn convolution kernels directly from images in 1989. LeNet-5, a seven-level network for digits, was fielded in several American banks in 1996, reading millions of checks per day.

Then the field stalled for a while on real photographs. The obstacles were data and compute. Two developments changed that. The first was ImageNet, a dataset begun by Fei-Fei Li in 2006 that grew past 14 million labeled images, with a 1,000-class, roughly 1.28-million-image subset used for the annual ILSVRC competition, which started in 2010 (ImageNet). The second was cheap graphics processors. Researchers showed GPU-trained CNNs running many times faster than on CPUs through the 2000s, a hardware story told in how GPUs work and why AI needs them.

The AlexNet moment, 2012

AlexNet, built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto, entered the ILSVRC on September 30, 2012. It had eight learned layers, five convolutional and three fully connected, about 60 million parameters, and was trained on two Nvidia GTX 580 GPUs with 3 GB of memory each, because the network did not fit on one card. Training took five to six days. Its top-5 error was 15.3 percent, more than 10.8 percentage points better than the runner-up (AlexNet).

The design combined several ingredients that are now standard:

• ReLU activations in place of sigmoid or tanh, which trained faster.
• Dropout, randomly silencing half of the units in the fully connected layers during training to reduce overfitting.
• Data augmentation, such as random crops and flips of the training images, to enlarge the effective dataset.
• Local response normalization, a neighbor-competition scheme later replaced by batch normalization.
• GPU training, which made all of the above practical.

The original paper reports 39.7 percent top-1 and 18.9 percent top-5 error on the 2010 challenge data, according to its abstract. The 2012 result drew attention because the margin was so large: the previous year's winner had scored about 25 percent, and the runner-up in 2012 was still near 26 percent.

The ImageNet years: how fast the error fell

The yearly winners tell the story compactly. The table uses top-5 error, meaning the correct label is missing from the network's five highest-probability guesses.

Year	Winning entry	Top-5 error	Note
2010	Pre-AlexNet winner	about 28%	Derived from 71.8% top-5 accuracy
2011	Pre-AlexNet winner	about 25%	Before deep learning arrived
2012	AlexNet	15.3%	First CNN winner
2014	GoogLeNet (Inception v1)	6.7%	22 layers; VGG took second in classification
2015	ResNet	3.57%	Residual connections
2017	SENet	2.251%	Latest winner listed in the ImageNet entry

Sources: the ImageNet entry for 2010, 2011, 2012, 2015, and 2017; Karpathy's analysis for GoogLeNet's 6.7 percent on the full test set.

How good is "human-level"?

The comparison point most often cited is 5.1 percent top-5 error for a human. That figure comes from a single experiment. Andrej Karpathy labeled 1,500 test images himself, after practicing on 500 validation images, and made mistakes on 5.1 percent of them. GoogLeNet scored about 6.8 percent on the same sample. He noted that a lot of his errors came from telling apart the 120-plus dog breeds among the 1,000 classes, and that less-practiced labmates got 12 to 15 percent. His summary was that human accuracy "is not a point" but a tradeoff curve.

So "AI beats humans at image recognition" is true only in a narrow, specific sense: on a closed set of 1,000 categories, scored by a particular metric, against one well-trained annotator. It does not say the machine understands pictures the way people do. The later section on failure modes returns to this.

Going deeper: VGG, Inception, and ResNet

The 2014 competition produced two influential designs, both exploring how to use depth.

VGG, from Oxford's Visual Geometry Group, made one bet: use only very small 3×3 filters and stack them 16 to 19 layers deep. Its authors reported placing first in localization and second in classification at the 2014 challenge, according to the paper's abstract. VGG-16 has 13 convolutional layers and 3 fully connected layers, about 138 million parameters, and training one network took two to three weeks on four GPUs (VGGNet). The design is simple and was widely reused, but it is expensive to run.

GoogLeNet, the Inception v1 network from Google, took a different route. It has 22 layers built from Inception modules, which apply several filter sizes in parallel and combine the results. Its authors describe the main hallmark as "improved utilization of the computing resources inside the network," allowing depth and width to grow "while keeping the computational budget constant" (Going Deeper with Convolutions). A key tool was the 1×1 convolution, which mixes information across channels at each position and cheaply reduces the number of channels before the costlier 3×3 and 5×5 filters.

ResNet and the skip connection

Both networks hit a wall at greater depth. In practice, simply stacking more layers on a plain network made accuracy worse, even on the training data. Kaiming He and colleagues called this the degradation problem, and the 2015 ResNet addressed it with a small change.

A residual block adds its input back to its output: instead of computing a new representation F(x), it computes F(x) + x. The shortcut path carries the input forward unchanged, so if the extra layers have nothing useful to add, they can learn to output near zero and the block passes its input through. As the Wikipedia article on residual networks puts it, stacking layers can degrade accuracy in plain networks, and identity shortcuts let extra layers default to doing nothing. The shortcut also gives gradients a direct path back to earlier layers during training. That article also notes that vanishing gradients are not the root cause of degradation, so a common shorthand that ResNet "fixed vanishing gradients" is only partly right.

ResNet won the 2015 challenge with 3.57 percent top-5 error. Its bottleneck blocks, built from 1×1, 3×3, and 1×1 convolutions, made ResNet-50, -101, and -152 practical, and variants with 200 to more than 1,000 layers have been trained. ResNet-50 remains a standard baseline for research and transfer learning. Residual connections also spread far beyond vision: they appear in Transformers, AlphaGo Zero, and AlphaFold, as the same article notes.

Batch normalization

One more ingredient made deep training smoother. Batch normalization, from Sergey Ioffe and Christian Szegedy, rescales activations within each mini-batch and then applies learned scale and shift parameters. Their abstract reports the same accuracy with 14 times fewer training steps on a state-of-the-art classifier, and an ensemble reaching 4.9 percent top-5 validation error, which they described as exceeding the accuracy of human raters (Batch Normalization).

How the major architectures compare

Architecture	Year	Depth	Distinctive idea	Notable figure
AlexNet	2012	8 learned layers	GPUs, ReLU, dropout	15.3% top-5 error, about 60M parameters
VGG-16	2014	16	Only 3×3 filters, stacked	About 138M parameters
GoogLeNet	2014	22	Parallel filter sizes, 1×1 reductions	6.7% top-5 error
ResNet-152	2015	152	Skip connections	3.57% top-5 error
EfficientNet-B7	2019	Scaled	Scale depth, width, and resolution together	84.3% top-1 accuracy
ConvNeXt	2022	Varies	ResNet modernized with Transformer-era design	87.8% top-1 accuracy

Beyond classification: what else CNNs do

Classification, assigning one label to a whole image, is the simplest task. The same convolutional backbone is reused for others. Object detection predicts boxes and labels for several objects. Segmentation labels every pixel. Image generation and style transfer run CNNs "in reverse" or in new arrangements, as in neural style transfer, and generative adversarial networks used convolutions heavily, as covered in how GANs work.

CNNs also underpin systems that must interpret camera input in real time. Self-driving stacks use convolutional feature extractors alongside other sensors, a topic explored in how self-driving cars work. In medicine, researchers apply them to scans and slides; the evidence and limits are discussed in AI cancer detection with deep learning.

Transfer learning

A network trained on ImageNet has learned general-purpose visual features, like edges, textures, and parts, that are useful far beyond its 1,000 categories. Zeiler and Fergus showed that retraining only the final classifier on an ImageNet-trained model beat earlier top results on the Caltech-101 and Caltech-256 datasets. The standard recipe today is to take a pretrained network, replace the last layer with one for the new classes, and train that layer, and sometimes a few more, on a small dataset. It cuts the data and compute needed sharply, which is why a team with a few thousand labeled images can still build a usable classifier.

EfficientNet, Vision Transformers, and where CNNs stand now

Once accuracy was high, the question became cost. EfficientNet (2019) argued that depth, width, and input resolution should be scaled together with one coefficient rather than one at a time. Its largest model, EfficientNet-B7, reached 84.3 percent top-1 accuracy on ImageNet while being, in the authors' words, "8.4x smaller and 6.1x faster on inference than the best existing ConvNet" (EfficientNet).

In 2020 the Vision Transformer (ViT) showed that a "pure transformer applied directly to sequences of image patches" could match state-of-the-art convolutional networks when pretrained on large datasets, and it needed fewer computing resources to train, according to the ViT abstract. The patches in the title are 16×16 pixels. Each patch is flattened into a vector and treated like a word, then processed with the attention mechanism described in how Transformers pay attention to words.

CNNs did not vanish. They have built-in assumptions, locality and weight sharing, that help when data is limited, while Transformers tend to need more data to learn such structure. In 2022 the ConvNeXt authors started with a standard ResNet and modernized it step by step, reaching 87.8 percent top-1 accuracy on ImageNet and competing favorably with Transformers (A ConvNet for the 2020s). The current practice is mixed: many production systems use CNNs for efficiency, many research systems use Transformers or hybrids, and diffusion-based image generators combine convolutional and attention layers, as in how diffusion models work.

Where image recognition still fails

High benchmark accuracy hides specific weaknesses. Two of them are well documented.

Texture bias. Robert Geirhos and colleagues tested networks on images where shape and texture disagree, such as a cat's shape filled with elephant skin. They reported that ImageNet-trained CNNs are "strongly biased towards recognising textures rather than shapes," unlike human observers. When the same ResNet-50 architecture was trained on a stylized version of ImageNet that removes texture cues, it learned a shape-based representation instead and was more robust to image distortions (ImageNet-trained CNNs are biased towards texture). The model is answering "what material is this" at least as much as "what shape is this."

Adversarial examples. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy showed that adding tiny, carefully computed perturbations to an image, invisible to people, can make a network confidently misclassify it. They argued that "the primary cause of neural networks' vulnerability to adversarial perturbation is their linear nature," which also explains why such examples transfer between models (Explaining and Harnessing Adversarial Examples). A picture of a panda that is relabeled as a gibbon after a faint noise pattern became the best-known illustration.

A third issue is distribution shift. ImageNet photos tend to be well lit and centered, and a model trained on them can stumble on unusual angles, lighting, or camera types. For practical systems, how you evaluate matters as much as the architecture: see why training and test data must be separated and why 99 percent accuracy can be useless. Overfitting, where a model memorizes its training set, is covered in overfitting and underfitting.

Common misconceptions

"A CNN looks at an image like a person does." It computes a stack of filter responses and a final score. It has no concept of an object's purpose, and the texture-bias results show it can rely on cues that people would call irrelevant.

"Deeper always means better." Plain deeper networks got worse before residual connections. Depth helps only when the architecture and training make it trainable.

"Pooling is what makes CNNs shift-proof." Convolution already makes feature maps shift along with the input. Pooling adds tolerance to small shifts but not large ones, and some modern designs reduce or drop pooling in favor of strided convolutions.

"Beating the 5 percent human error means machines see better than people." As Karpathy's experiment shows, the human number depends on training and effort, the label set is closed, and many of the human errors were in distinguishing similar dog breeds. The benchmark is a useful yardstick, not a verdict.

FAQ

Q: How do CNNs handle images of different sizes? A: Most classification networks are trained on fixed-size inputs, such as 224×224, so images are resized or cropped before use. Convolutional layers themselves work on any size. Networks that end in global average pooling instead of a fixed fully connected layer can accept variable-size inputs.

Q: What is a feature map? A: A feature map is the output of one filter applied across an entire image. It is a grid showing where that filter's pattern was detected and how strongly. A layer with 64 filters produces 64 feature maps, and the next layer treats that stack as its input.

Q: What is transfer learning for CNNs? A: It means reusing a network trained on a large dataset, usually ImageNet, for a new task. You replace the final classification layer and train it, and often a few later layers, on your smaller dataset. Early layers capture general edges and textures, so they transfer well.

Q: Why is ImageNet accuracy not the full story? A: ImageNet is a closed set of 1,000 categories with mostly clear, centered photos. Models that score well can still depend on texture over shape, and can fail under unusual lighting, viewpoints, or small adversarial perturbations. Real deployments need testing on data that resembles real use.

Q: What is batch normalization and why does it matter? A: Batch normalization rescales the activations in each mini-batch to a standard range, then applies learned scale and shift parameters. Its authors reported matching accuracy with 14 times fewer training steps on one model. It is used in most modern CNNs because it makes training faster and more stable.

Q: Can CNNs be fooled? A: Yes. Adversarial examples are small, deliberately computed changes to an image, often invisible to people, that make a network confidently pick the wrong label. Research suggests the networks' near-linear behavior in high dimensions is a major cause, and defenses reduce but do not eliminate the problem.

Q: Have Vision Transformers replaced CNNs? A: Not entirely. Vision Transformers match or beat CNNs when pretrained on very large datasets, but CNNs have useful built-in assumptions and remain efficient, especially with limited data or on small devices. Modernized designs such as ConvNeXt show that convolutions remain competitive.

The takeaway

The core of image recognition is a bargain between structure and learning. Convolution builds in two assumptions, that patterns are local and that the same pattern can appear anywhere, and then lets training discover which patterns matter. That bargain cut the parameter count enough to make photographs tractable, and stacking layers turned simple detectors into a hierarchy that ends in object categories.

What changed between 2010 and 2015 was not one invention but a convergence: a large labeled dataset, GPUs, and a run of architectural fixes culminating in skip connections. What has not changed is the gap between scoring well on a benchmark and seeing the way people do. The most useful habit when reading any claim about machine vision is to ask what the test set was, what the metric rewarded, and what the model might be using as a shortcut.

Sources

• Krizhevsky, Sutskever, Hinton, ImageNet Classification with Deep Convolutional Neural Networks (NeurIPS 2012)
• Zeiler and Fergus, Visualizing and Understanding Convolutional Networks
• Simonyan and Zisserman, Very Deep Convolutional Networks
• Szegedy et al., Going Deeper with Convolutions
• Ioffe and Szegedy, Batch Normalization
• Tan and Le, EfficientNet
• Dosovitskiy et al., An Image Is Worth 16x16 Words
• Liu et al., A ConvNet for the 2020s
• Geirhos et al., ImageNet-trained CNNs are biased towards texture
• Goodfellow, Shlens, Szegedy, Explaining and Harnessing Adversarial Examples
• Karpathy, What I learned from competing against a ConvNet on ImageNet
• Wikipedia: ImageNet, AlexNet, Convolutional neural network, VGGNet, Residual neural network

---

Read next:

• How Large Language Models Work: Transformers, Attention, and Why They Hallucinate
• AI in Healthcare: Transforming Diagnosis and Treatment
• Transformers Explained: The Architecture Behind GPT