Computer Vision AI: How Machines See

A radiologist glances at a chest X-ray and notices a suspicious shadow. A self-driving car registers a cyclist cutting across three lanes. A factory camera catches a hairline crack in a metal casting before the part ships. In each case, something is turning raw pixels into meaning, and the biological version of that trick took hundreds of millions of years of evolution to build.

Computer vision is the field of AI that gives machines a version of it. Calling it "seeing" is slightly misleading, though. A camera records photons as numbers, and everything a vision system does afterward is arithmetic on those numbers. This article follows that arithmetic: the handcrafted methods that came first, the 2012 result that changed the field, how convolutional networks and Vision Transformers build understanding, and the places where machine vision still breaks in ways a toddler's does not.

---

What a computer actually receives

A digital photograph is a grid of numbers. A modest 224 by 224 pixel color image holds 224 × 224 × 3 = 150,528 values, one each for the red, green, and blue intensity of every pixel, usually stored as integers from 0 to 255. There is no cat in that grid, no stop sign, no tumor. There is only a very large table, and the task is to map that table to something a person would call a description.

That mapping is hard because the same object produces wildly different tables. Move the camera two steps left and almost every number changes. Dim the lights and every value shrinks. Let a coffee cup partly hide the cat and a chunk of the grid is filled with cup instead. A useful vision system has to ignore all of that variation while staying sensitive to the small differences that separate a cat from a fox. Researchers call the first property invariance and the second discrimination, and nearly every design decision in the field is a trade between them.

---

Before deep learning: the era of handcrafted features

For most of computer vision's history, roughly the 1960s until 2012, researchers tried to teach machines to see by writing rules. The strategy was intuitive. Decide which measurable properties of an image are likely to matter, compute them by formula, and hand them to a classifier.

Three techniques from this period are worth knowing, because the way they failed explains why the field changed course.

• Viola-Jones face detection (2001). Paul Viola and Michael Jones built a cascade of simple classifiers that rejects a candidate window as soon as one stage decides it contains no face. Early stages are cheap and throw out most of the image, so the expensive tests run on very few windows. Wikipedia's summary of the method reports about 15 frames per second on 384 × 288 images with a 700 MHz Pentium III, which was real time on 2001 hardware.
• HOG, the Histogram of Oriented Gradients. Navneet Dalal and Bill Triggs presented the version that became standard at CVPR in 2005, focused on finding pedestrians in still images. HOG measures the direction of brightness changes in small cells, pools them into histograms, normalizes across neighboring blocks to cancel out lighting, and feeds the resulting vector to a classifier. Their design choices included nine orientation bins per cell, which worked best for people.
• SIFT, the Scale-Invariant Feature Transform. David Lowe's method (introduced in 1999, refined in a 2004 journal paper) finds distinctive local keypoints and describes each with a 128-number vector built from orientation histograms. The description stays similar when an image is scaled or rotated, so SIFT can match the same building across two photographs taken from different places, per the SIFT overview.

The classifier at the end of these pipelines was typically a Support Vector Machine (SVM), an algorithm that finds the boundary separating two groups of feature vectors with the widest possible margin.

These systems worked well on narrow problems, but they were brittle. Every feature encoded a researcher's guess about what matters, and guesses fail at the edges. A HOG pedestrian detector tuned for upright walkers struggled with someone crouching. SIFT matched textured surfaces beautifully and did poorly on smooth ones. Each new task meant a new round of feature design, and progress on the standard benchmarks crept forward in fractions of a percent per year. What the field wanted was a way to let the machine discover its own features from data.

---

Borrowed from the brain: where convolution came from

The idea that eventually delivered that came from neuroscience. In the 1950s and 1960s, David Hubel and Torsten Wiesel recorded from neurons in a cat's visual cortex and found that each cell responds only to a small patch of the visual field, and that some cells prefer edges at particular angles. This patch is the cell's receptive field. The history of the convolutional neural network traces a direct line from those recordings to Kunihiko Fukushima's 1980 neocognitron, which stacked layers of shared-weight feature detectors and downsampling layers. Our article on Hubel and Wiesel's visual cortex experiments covers the original work.

Fukushima's network could not be trained efficiently. In 1989, Yann LeCun and colleagues used backpropagation, the algorithm that adjusts weights by following the error backward through the network, to learn convolution kernels directly from images of handwritten digits. By 1995 this had matured into LeNet-5, a seven-level network that read handwritten amounts on checks and, according to the same history, was fielded in several American banks from June 1996.

So convolutional networks existed for more than two decades before they took over. What they lacked was enough data, enough compute, and a few training tricks. All three arrived around 2010.

---

AlexNet 2012: the watershed moment

ImageNet, the project Fei-Fei Li began in 2006 and presented at CVPR in 2009, grew into a collection of roughly 14 million labeled images. Starting in 2010, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) asked teams to classify images into 1,000 categories, using a training set of 1,281,167 images, and scored them on top-5 error: the fraction of images where the correct label is not among the model's five best guesses.

In 2010 and 2011, the winners were built from the handcrafted-feature tradition and landed at about 28% and 25% top-5 error. In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton entered a deep convolutional network, later known as AlexNet, and cut the error to 15.3%, more than 10 percentage points ahead of the runner-up (ImageNet and AlexNet entries; the original paper is Krizhevsky et al., 2012). Year-over-year gains on this benchmark had been measured in single points. This was a jump of about ten.

Year	Winning approach	Top-5 error	Note
2010	Linear SVM on handcrafted features	about 28%	Pre-deep-learning
2011	XRCE, handcrafted features	about 25%	Last non-neural winner
2012	AlexNet	15.3%	60 million parameters, five convolutional layers
2015	ResNet (ensemble)	3.57%	Up to 152 layers
2017	SENet	2.25%	Final ILSVRC year

For comparison, Andrej Karpathy, who labeled a sample of ImageNet validation images himself, estimated a human error of about 5.1% when working with concentrated effort. Machines passed that mark with ResNet in 2015, on this narrow and carefully defined task.

AlexNet worked because of four bets that paid off together.

1. GPU training. The network did not fit in the memory of one graphics card, so it was split across two Nvidia GTX 580 GPUs with 3 GB of memory each. Training took five to six days. Graphics hardware turned out to be ideally suited to the huge number of parallel multiplications a convolutional network requires, a fact that why AI needs so much computing power explores in more depth.
2. ReLU activations. The Rectified Linear Unit outputs its input if positive and zero otherwise. Unlike the older sigmoid and tanh functions, it does not flatten out for large inputs, so gradients keep flowing and training goes much faster.
3. Dropout. During training, each neuron in the large fully connected layers was switched off at random with probability 0.5, which prevented neurons from relying on one another and reduced overfitting.
4. Data augmentation. Random 224 × 224 crops and horizontal flips of the training images expanded the effective dataset by a factor of about 2,048, without anyone labeling a new photo.

The 2012 entry was also an ensemble: seven AlexNets whose predicted probabilities were averaged. Within two years, almost every serious entry in the challenge was a deep convolutional network. Machine learning research did not reorganize itself overnight, but it did so quickly enough that 2012 is now treated as the start of the modern deep learning era.

---

How a CNN computes: filters, feature maps, and a hierarchy

To see why convolutional networks work, it helps to do the arithmetic once by hand.

A convolutional filter (or kernel) is a small grid of learnable numbers, commonly 3 × 3. Place it over a 3 × 3 patch of the image, multiply each filter weight by the pixel beneath it, and add the nine products. That single number is the filter's response at that location. Slide the filter one pixel at a time across the whole image and you get a new grid, a feature map, where each value says how strongly that pattern appears at that position.

Here is a concrete case. A filter with the rows (-1, 0, 1), (-2, 0, 2), (-1, 0, 1) is a classic vertical-edge detector. Over a flat gray patch it outputs zero, because the negative and positive weights cancel. Over a patch where the left column is dark (value 0) and the right column is bright (value 200), every row contributes, and the response is 200 × 1 + 200 × 2 + 200 × 1 = 800. The filter lights up at vertical edges and stays quiet elsewhere. In a trained network nobody writes that filter. It emerges from training, because detecting edges turns out to be useful for lowering the error.

Why convolution saves so many parameters

The reason convolution beats a plain fully connected layer is weight sharing. A fully connected layer that connects a 224 × 224 × 3 image (150,528 inputs) to just 1,000 neurons needs about 150.5 million weights. A convolutional layer with 128 filters of size 3 × 3 over 64 input channels needs 3 × 3 × 64 × 128 = 73,728 weights, and it applies the same filters at every position. The network does not have to relearn "edge" separately for the top-left and bottom-right corners of the picture.

This matches how convolutional layers mimic the receptive fields Hubel and Wiesel found: each unit looks at a restricted neighborhood, and stacking layers widens what each unit can see. Two stacked 3 × 3 layers cover a 5 × 5 region of the original image, three cover 7 × 7, and so on. Deep networks are, in effect, a way to look at ever larger context while only ever computing small local operations.

What the layers learn

In 2013, Matthew Zeiler and Rob Fergus published a visualization technique that showed what individual layers of an ImageNet-trained network respond to (paper, presented at ECCV 2014). Their images confirmed the hierarchy researchers had suspected.

Depth	What units respond to	Rough analogy
Layers 1 and 2	Edges, corners, color gradients	Pen strokes
Layers 3 and 4	Textures, repeating patterns, simple shapes	Syllables
Layers 5 and beyond	Object parts such as wheels, eyes, fur, text	Words
Final layers	Combinations of parts that signal a whole category	Sentences

No engineer wrote "look for wheel-shaped things." The network found that wheels are a useful intermediate representation for recognizing cars and organized itself accordingly. This is the central difference from the handcrafted era: the features are learned along with the classifier, so the system can build exactly the vocabulary the task needs.

Max pooling layers sit between convolutional layers and keep only the strongest activation in each small region, typically every 2 × 2 block. That halves the width and height of the feature map, cuts computation, and gives the network some tolerance for small shifts: a cat that moves a few pixels still triggers the same pooled response. At the end of the network, the final feature maps are flattened into a vector and passed through fully connected layers that output one probability per class.

---

How a network learns to see

Training a vision model follows the same recipe as any supervised neural network, described in more detail in our article on gradient descent. The network starts with random weights, guesses a label for each training image, and a loss function measures how wrong the guess was. For classification the usual loss is cross-entropy, which punishes confident wrong answers heavily. Backpropagation then computes how much each of the millions of weights contributed to the error, and gradient descent nudges every weight a small step in the direction that reduces it. Repeat over the full dataset dozens of times and the filters settle into useful shapes.

Two practical problems dominate. The first is overfitting: with 60 million parameters and 1.28 million images, a network can memorize training examples instead of learning general features. Dropout and augmentation, the two tricks from AlexNet, remain standard defenses, and why overfitting hurts machine learning explains the underlying trade-off. The second is data. Collecting and labeling a million images is expensive, which is why transfer learning became the default workflow. A network pretrained on ImageNet or a larger web collection has already learned generic edges, textures, and parts. Fine-tuning that network on a few thousand labeled images from a new domain, such as factory parts or skin photographs, is far cheaper than training from scratch.

---

One field, many tasks

Classification asks what is in the picture. Most real applications need more than that, and different questions need different outputs.

Task	Output	Example models	Core challenge
Image classification	One label per image	AlexNet, ResNet, EfficientNet	Generalizing across viewpoints and lighting
Object detection	Bounding boxes plus labels	YOLO, SSD, Faster R-CNN	Locating several objects at different scales
Instance segmentation	A pixel mask for each object	Mask R-CNN, SAM	Separating overlapping objects
Semantic segmentation	A class label for every pixel	DeepLab, SegFormer	Dense, cluttered scenes
Pose estimation	Coordinates of body joints	OpenPose, MediaPipe	Occlusion and unusual poses
Depth estimation	A distance estimate per pixel	DPT, ZoeDepth	Recovering 3D from a 2D image
Image generation	A new image from a prompt	Stable Diffusion, DALL-E	Coherence and consistency

Object detection and the speed-accuracy trade

Object detection is where computer vision becomes economically significant, because it answers "what is where." Detectors are scored with intersection over union (IoU): the area where a predicted box overlaps the true box divided by the area they cover together. An IoU of 1 is a perfect match, and a common convention counts a detection as correct above 0.5.

Two families dominate. Faster R-CNN (Ren, He, Girshick, and Sun, 2015) added a Region Proposal Network that shares convolutional features with the detector, which makes proposing candidate regions almost free. With a VGG-16 backbone it ran at about 5 frames per second on a GPU (paper). YOLO ("You Only Look Once," Redmon and colleagues, 2015) took a different route: treat detection as a single regression problem and predict boxes and class probabilities for the whole image in one forward pass. The base model reached about 45 frames per second, and a smaller Fast YOLO variant reached about 155 (paper). A system that must keep up with a 60 frame-per-second camera on a production line leans toward the YOLO design. A medical system that reads archived scans offline can afford a slower, more careful detector. The practical side of this is covered in computer vision applications in 2026.

Segmentation and the "segment anything" shift

Segmentation assigns labels at the pixel level, which is what a radiation oncologist needs when outlining a tumor. In 2023, Meta AI released the Segment Anything Model (SAM) and its dataset SA-1B, with more than 1 billion masks on 11 million images. SAM is promptable: a user clicks a point or draws a box and the model returns the object's outline, with no task-specific training. The authors report that its zero-shot performance is often competitive with, or better than, earlier fully supervised results (paper). Task-specific training data used to be the price of entry for a segmentation tool, and SAM showed that a broad enough pretrained model can cover many tasks without it.

For a closer look at CNN mechanics beyond what fits here, see CNNs explained: how image classification actually works.

---

Modern architectures: ResNet, EfficientNet, and Vision Transformers

ResNet and the skip connection

Stacking more layers should let a network learn richer features, but in practice very deep plain networks became harder to train and sometimes performed worse than shallower ones. In 2015, Kaiming He and colleagues at Microsoft Research proposed residual connections: a shortcut that adds a block's input directly to its output, so each block only has to learn a correction to what it received. Gradients can travel back along the shortcut without being squeezed by every layer in between. The resulting networks reached up to 152 layers, eight times deeper than the VGG networks, with lower complexity, and an ensemble won ILSVRC 2015 with 3.57% error (paper). The same work reported a 28% relative improvement on the COCO detection dataset. ResNet-50, a 50-layer member of the family, is still one of the most common default backbones in applied work.

EfficientNet and compound scaling

Making a network better usually means making it bigger, and there are three ways to do that: more layers (depth), more channels per layer (width), or higher-resolution input images. Mingxing Tan and Quoc Le noticed that scaling only one of them saturates quickly and proposed scaling all three together by a fixed ratio. Their largest model, EfficientNet-B7, reached 84.3% top-1 accuracy on ImageNet while being 8.4 times smaller and 6.1 times faster at inference than the best existing convolutional network they compared against (paper).

Vision Transformers

Transformers took over language modeling in the late 2010s, and in 2020 Alexey Dosovitskiy and colleagues showed that the same architecture works on images with almost no changes. A Vision Transformer (ViT) cuts the image into fixed-size patches, flattens each one into a vector, and feeds the sequence of vectors through self-attention layers, exactly as a language model processes words. The paper's title, "An Image is Worth 16x16 Words," refers to the 16 × 16 pixel patches. On a 224 × 224 image that gives 196 patches, so self-attention compares 196 × 196 = 38,416 pairs of patches at every layer. For the attention mechanism itself, see how transformers pay attention to words and Transformers explained simply.

The authors found that when pretrained on large datasets and transferred to smaller benchmarks, ViT compares favorably with state-of-the-art convolutional networks while needing substantially fewer computational resources to train (paper). There is a catch in the phrase "pretrained on large datasets." A CNN has local structure built in through its small filters, so it starts with a sensible assumption about images. A ViT has to learn that nearby patches tend to matter to one another, and it needs more data to do so.

Architecture	Core idea	Strength	Trade-off
CNN (AlexNet, VGG)	Local filters with shared weights	Efficient, strong with modest data	Limited global context in early layers
ResNet	CNN plus skip connections	Trains to 100+ layers	Still local at each step
EfficientNet	Scale depth, width, and resolution together	High accuracy per parameter	Tuned for a fixed compute budget
Vision Transformer	Self-attention over image patches	Global context from layer one; scales well	Data-hungry when trained from scratch

---

The self-supervised turn

One bottleneck has always been labels. ImageNet needed an army of annotators, and medical datasets need specialists to mark every scan. Self-supervised learning removes the human by building the training signal out of the data itself.

DINO (Caron and colleagues, 2021) trains a Vision Transformer to produce consistent representations for different crops of the same image, with no labels at all. The authors found that the resulting features contain explicit information about object segmentation that does not emerge as clearly in supervised ViTs or in convolutional networks, and a small model reached 78.3% top-1 accuracy on ImageNet with a simple nearest-neighbor classifier (paper). DINOv2 (Oquab and colleagues, 2023) scaled the approach to a one-billion-parameter ViT trained on an automatically curated dataset, then distilled it into smaller models that outperform OpenCLIP on most benchmarks (paper).

Masked Autoencoders (MAE) use a different puzzle. The model hides 75% of an image's patches and learns to reconstruct what is missing, which forces it to pick up the structure of objects and scenes. A ViT-Huge pretrained this way reached 87.8% accuracy on ImageNet-1K, the best result among methods using only that dataset when the paper appeared (paper). Think of it as a student who learns what a face looks like by being asked to complete a mostly torn-up photograph.

CLIP (Contrastive Language-Image Pretraining, OpenAI, 2021) links vision to language. It trains an image encoder and a text encoder together on 400 million image-text pairs collected from the internet, pulling each image's embedding toward the embedding of its caption and pushing mismatches apart. The payoff is zero-shot classification: to recognize a new category you simply write its name as text. The authors report that CLIP matches the accuracy of the original ResNet-50 on ImageNet without using any of the 1.28 million training examples that ResNet-50 saw, and they benchmarked it on more than 30 other vision datasets (paper). The shared embedding space is the same idea that powers embeddings and vector search.

---

Where computer vision works in practice

Medical imaging

Medical imaging has produced the most carefully measured results. In 2016, a Google team led by Varun Gulshan trained a deep network on 128,175 retinal photographs, each graded by a panel of ophthalmologists, to detect referable diabetic retinopathy. At a high-specificity operating point it reached 90.3% sensitivity and 98.1% specificity on one validation set of 8,788 gradable images and 87.0% and 98.5% on a second (JAMA). Sensitivity is the share of true cases caught, and specificity is the share of healthy patients correctly cleared.

In April 2018, the US Food and Drug Administration authorized IDx-DR, the first device cleared to give a screening decision for more than mild diabetic retinopathy without a clinician also interpreting the images. The pivotal study enrolled 900 patients with diabetes at 10 primary care sites, and the software correctly identified disease 87.4% of the time and correctly cleared patients without it 89.5% of the time (report on the FDA announcement). Those numbers also show the limit: roughly one in ten patients got a wrong answer in each direction, which is why such tools are designed as screening aids, not as replacements for follow-up care. Our article on AI cancer detection with deep learning looks at similar work in oncology.

Facial recognition

Face recognition systems embed each face into a high-dimensional space where images of the same person cluster together and images of different people sit apart, then compare distances. Whether they work equally well for everyone is an empirical question, and the largest public test says not always. In its December 2019 study (NISTIR 8280), the US National Institute of Standards and Technology evaluated 189 algorithms from 99 developers on 18.27 million images of 8.49 million people. For one-to-one matching, false positive rates for Asian and African American faces were often 10 to 100 times higher than for Caucasian faces, depending on the algorithm (NIST). The size of the gap varied widely between algorithms, which suggests that training data and design choices matter and that the problem is fixable in part, but it also means a vendor's headline accuracy figure tells you little without the demographic breakdown.

Vehicles, factories, and satellites

Autonomous driving stacks combine camera images with LiDAR, which measures distance by timing laser pulses (see how LiDAR works). Cameras provide rich semantic detail such as sign text and traffic-light color, and LiDAR provides precise depth. The fusion software tracks pedestrians, estimates where other vehicles are heading, and updates a model of free space many times each second. The full pipeline, and why it remains difficult, is described in how self-driving cars actually work.

On factory lines, machine vision inspects parts at line speed, catching surface scratches, dimensional errors, and misaligned labels more consistently than a person on the eighth hour of a shift. Satellite and aerial imagery analysis tracks deforestation, crop condition, and new construction across areas no human team could review manually. In each case the economics are the same: a vision model does not need to be perfect, only cheaper and steadier than the alternative, with humans reviewing the flagged cases.

---

How machine vision fails differently from human vision

A network that scores well on ImageNet looks intelligent, and then someone changes a few pixels.

Adversarial examples

The best-known demonstration comes from Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. In their 2014 paper, an image that GoogLeNet labeled "panda" with 57.7% confidence was altered by adding a tiny, carefully computed perturbation, and the network then called it a "gibbon" with 99.3% confidence. The perturbation's size, ε = 0.007, corresponds to the smallest bit of an 8-bit image encoding, which makes it invisible to people (paper). The authors argued the cause is partly the nearly linear behavior of networks in high dimensions: many tiny changes, each aimed in the worst direction, add up to a large change in the output.

The panda example is a digital attack, with direct access to the model's pixels. Even so, it shows that a network's decision boundary can sit very close to ordinary-looking images, which is why teams building safety-critical systems test against adversarial inputs deliberately.

Texture bias

A second difference concerns what the model attends to. In a 2019 ICLR paper, Robert Geirhos and colleagues tested ImageNet-trained CNNs on images where texture and shape point to different answers. The networks leaned heavily on texture, while human observers relied on shape. When the team trained a ResNet-50 on Stylized-ImageNet, a version of the dataset with the textures repainted, the network learned a shape-based representation that matched human behavior much more closely across experiments involving 97 observers and 48,560 trials, and it also became more robust to image distortions (paper).

A texture-biased medical model can learn the wrong lesson from a scanner. If images from one hospital carry a subtle signature from its equipment and that hospital also has more sick patients, the network may learn the signature instead of the disease, then fail when it meets another hospital's images. This is the kind of failure that model drift in production describes from the engineering side, and it is one reason evaluation on data from new sites matters as much as headline accuracy.

A short list of other limits

• Distribution shift. A model trained on stock photography can stumble on sketches, night images, or unusual camera angles.
• No common sense. A detector can find a "person" printed on a billboard and has no reason to doubt it.
• Label noise. Training labels written by tired annotators contain errors, and big models partly memorize them.
• Calibration. Confidence scores are often higher than the actual probability of being right, as the 99.3% panda-to-gibbon example shows.

---

Generative AI: vision in reverse

Classification takes an image and returns a description. Generative models run the arrow the other way. Diffusion models, the architecture behind Stable Diffusion and DALL-E 3, learn from training images that have been progressively corrupted with noise, then learn to undo the corruption one small step at a time. At generation time they start from pure noise and denoise toward an image, steered by a text prompt encoded with a CLIP-like model. We explain the process in how diffusion models work. Earlier generative adversarial networks (GANs) pit a generator against a discriminator, a design covered in how GANs work. Image-to-image models extend the idea, taking an input image plus an instruction and returning an edited version.

Video models such as Sora apply similar ideas across time, trying to keep objects consistent from frame to frame. Whether any generative model understands what it draws is an open debate. For the purposes of this article, the point is that recognition and generation increasingly share parts: the CLIP text encoder that helps classify an image also helps steer one into existence.

---

The takeaway

Computer vision did not become good at images because someone finally wrote better rules. It became good when researchers stopped writing the rules, let convolutional filters be learned from over a million labeled images, and gave the training process enough GPU time to find them. Every advance since, from skip connections to Vision Transformers to self-supervised pretraining, has been a way of getting more learning out of less human effort.

The failures follow from the same fact. A model that discovers its own features will discover whichever features lower its training error, including textures, scanner artifacts, and statistical quirks no person would choose. That is why the strongest evidence for a vision system is rarely its benchmark score and almost always a test on data it has never seen, ideally from a different place and a different population than the one it was trained on. For related background, see neural networks and deep learning and the complete AI guide.

---

Frequently Asked Questions

Q: What is the difference between computer vision and image processing? A: Image processing transforms pixels, for example by sharpening, denoising, or adjusting color, without trying to understand what the picture shows. Computer vision aims to extract meaning: which objects are present, where they are, and what is happening. Modern vision systems often use image processing as a preprocessing step, but the goal is interpretation.

Q: Do CNNs need huge amounts of labeled data? A: Training from scratch does. AlexNet learned from 1.28 million labeled images. Transfer learning changes the picture: starting from a model pretrained on a large dataset, a team can fine-tune with hundreds or thousands of labeled examples. Self-supervised models such as DINO and MAE cut label requirements further, since they learn from unlabeled images.

Q: Why did AlexNet's 2012 ImageNet result matter so much? A: It lowered top-5 error to 15.3% when the previous winners were at about 25%, a gap of more than 10 points in a benchmark where yearly gains had been small. It also showed that one particular recipe, deep convolutional networks trained on GPUs with ReLU, dropout, and data augmentation, was a different kind of approach from the handcrafted pipelines it displaced.

Q: How does a Vision Transformer differ from a CNN? A: A CNN slides small learnable filters over the image, so each unit sees a local neighborhood and context grows with depth. A Vision Transformer splits the image into patches, such as 196 patches of 16 × 16 pixels for a 224 × 224 image, and lets every patch attend to every other patch from the first layer. ViTs generally need more pretraining data, but they scale well and often match or beat CNNs when that data is available.

Q: What are adversarial examples, and should I worry about them? A: They are inputs altered, often imperceptibly, so that a model misclassifies them. In the best-known example, a panda image altered at the scale of one bit of 8-bit color was labeled "gibbon" with 99.3% confidence. For a photo-tagging app the risk is small. For safety-critical uses such as driving or medical screening, developers test for them deliberately and do not rely on benchmark accuracy alone.

Q: How does CLIP connect images and text? A: CLIP trains an image encoder and a text encoder together on 400 million image-caption pairs. The training objective pulls matching image and caption embeddings together and pushes mismatches apart, so both end up in one shared space. To classify an image, you embed candidate category names as text and pick the one closest to the image embedding, with no task-specific training.

Q: Is computer vision the same as seeing the way humans do? A: No. Vision models run learned statistical transformations on pixel arrays. Human vision combines retinal signals with memory, attention, and expectations about how the world works. Networks show a texture bias where people show a shape bias, and they can be fooled by perturbations people never notice. "Seeing" is a convenient shorthand for a different mechanism.

---

Sources

• Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25.
• He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep residual learning for image recognition. CVPR 2016.
• Zeiler, M. D., and Fergus, R. (2013). Visualizing and understanding convolutional networks. ECCV 2014.
• Goodfellow, I. J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples.
• Geirhos, R., et al. (2019). ImageNet-trained CNNs are biased towards texture. ICLR 2019.
• Tan, M., and Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. ICML 2019.
• Dosovitskiy, A., et al. (2020). An image is worth 16x16 words. ICLR 2021.
• Radford, A., et al. (2021). Learning transferable visual models from natural language supervision. ICML 2021.
• Caron, M., et al. (2021). Emerging properties in self-supervised vision transformers. ICCV 2021.
• He, K., et al. (2021). Masked autoencoders are scalable vision learners.
• Oquab, M., et al. (2023). DINOv2: Learning robust visual features without supervision.
• Kirillov, A., et al. (2023). Segment anything.
• Redmon, J., et al. (2015). You only look once: Unified, real-time object detection.
• Ren, S., et al. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks.
• Gulshan, V., et al. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA.
• National Institute of Standards and Technology (2019). NIST study evaluates effects of race, age and sex on face recognition software.
• Background references: ImageNet, AlexNet, Convolutional neural network, Viola-Jones, HOG, SIFT.

---

Read next:

• How GPUs Work: Why AI Needs Them
• Why 99% Accuracy Can Be Useless: Precision and Recall
• AI Image Generators Explained