Computer Vision Object Detection Explained: From YOLO to Vision Transformers

Object detection is the task behind a car noticing a pedestrian, a phone focusing on a face, and a warehouse robot spotting one package among hundreds. Unlike plain image classification, a detector has to say what is in the picture and where, usually as a labeled rectangle around each object, and often fast enough to keep up with video. This article follows the technique from the R-CNN family through YOLO to transformer detectors, and explains how each is scored, so benchmark claims become readable.

Detection answers three questions at once

Image classification asks one question about a whole picture: is this a cat? Object detection asks three about every object: what is it, where exactly is it, and how many are there? The output is a list of detections, each with a class label, a bounding box (four numbers giving the rectangle's position and size), and a confidence score between 0 and 1.

The difference matters whenever location drives a decision. Knowing that "a pedestrian is somewhere in this frame" tells a car nothing useful. Knowing the pedestrian sits 15 meters ahead and slightly left, and was in a different spot a tenth of a second ago, is the input a braking system needs. A neighboring task, instance segmentation, goes one step further and outputs a pixel-accurate outline instead of a rectangle. Segmentation costs more computation but is needed for jobs where shape matters, such as surgical tools or augmented reality.

A useful analogy: classification is like glancing at a bookshelf and saying "there are books here." Detection is like a librarian walking the shelf and writing down every title along with its position, including spines that overlap and books half hidden behind a plant. Much of the history of the field is about making that walk faster without making the librarian sloppier.

How detectors are scored: IoU, precision, and mAP

Before comparing models, it helps to know what the numbers measure. Almost every detection paper reports mean average precision, or mAP, and it is built from a simple geometric idea.

Intersection over Union

Intersection over Union (IoU) measures how well a predicted box overlaps the true box: the area they share divided by the total area they cover together. Take two 100 by 100 pixel boxes. If the prediction is shifted 20 pixels sideways, the overlap is 80 by 100, or 8,000 square pixels, and the combined area is 20,000 minus 8,000, or 12,000. The IoU is about 0.67. Shift it 50 pixels and the overlap is 5,000, the union is 15,000, and the IoU falls to about 0.33.

A detection counts as correct when its IoU with a ground-truth box passes a threshold, commonly 0.5, and its class label is right. Duplicate detections of the same object count as errors, which is why post-processing matters later in this article.

Precision, recall, and average precision

Precision is the fraction of detections that are correct. Recall is the fraction of real objects that were found. Raise the confidence cutoff and precision goes up while recall goes down. Plot that tradeoff as a curve, and the area under it is the average precision (AP) for one class. mAP averages AP across classes. The same logic underlies the general lesson in why 99% accuracy can be useless.

The COCO benchmark's headline metric, usually written AP or mAP@[.5:.95], goes one step further and averages over ten IoU thresholds from 0.5 to 0.95. A detector that draws loose boxes can score well at 0.5 and poorly at 0.9, so this average rewards tight localization. COCO also reports AP separately for small, medium, and large objects, which exposes a weakness many detectors share.

What COCO contains

COCO, short for Common Objects in Context, is the standard benchmark. Its 2014 paper describes 2.5 million labeled instances in 328,000 images across 91 object types (Lin et al., arXiv:1405.0312). The detection challenge uses a subset of 80 classes. According to the Ultralytics dataset documentation, the standard training split has 118,287 images, with 5,000 validation images and about 20,000 test images whose labels are held back on an evaluation server.

Benchmark fact	Value	Source
Original COCO paper scale	2.5 million instances, 328,000 images, 91 types	arXiv:1405.0312
Detection classes in the standard task	80	Ultralytics COCO docs
Train2017 images	118,287	Ultralytics COCO docs
Val2017 images	5,000	Ultralytics COCO docs

One caution carries through the rest of the article. A mAP number is only comparable when the dataset, the image size, the threshold averaging, and the evaluation split match.

Before deep learning: the sliding window problem

The earliest practical approach was brute force. Train a classifier on small crops, then slide it over every position of the image at several sizes, and keep the windows where it fires. The cost adds up quickly. On a 640 by 480 image, a 64 by 64 window moved in 8-pixel steps visits 73 columns and 53 rows, or 3,869 positions, at one scale. Multiply by a dozen window sizes and you are running a classifier tens of thousands of times for a single picture.

The idea was workable with fast, hand-designed classifiers and impractical with deep networks, which cost far more per crop. Deep learning arrived in this field after AlexNet's 2012 win on the ImageNet classification challenge showed that convolutional networks could learn image features better than hand-built ones. The question became how to point that power at the where question without running it a million times.

Two-stage detectors: propose, then classify

The R-CNN answer was to be selective about where to look. R-CNN, short for Regions with CNN features, by Girshick, Donahue, Darrell, and Malik, generated roughly two thousand candidate regions with a separate algorithm, resized each, and ran a convolutional network on each one. It was posted in late 2013 and reported a mean average precision of 53.3 percent on PASCAL VOC 2012, more than 30 percent better in relative terms than the previous best result (arXiv:1311.2524). The method was accurate and painfully slow, because the network ran from scratch on every region.

Sharing the work

Fast R-CNN fixed the redundancy. It ran the convolutional network once on the whole image, then pulled features for each candidate region from the shared feature map. Its authors report training a very deep VGG16 network 9 times faster than R-CNN and testing 213 times faster (arXiv:1504.08083).

Faster R-CNN then removed the separate proposal algorithm, which had become the bottleneck. It added a Region Proposal Network (RPN), a small fully convolutional network that shares features with the detector and proposes regions at almost no extra cost. With VGG-16 the whole system ran at about 5 frames per second on a GPU, using only 300 proposals per image (arXiv:1506.01497).

Two-stage designs remain strong where accuracy and precise boxes matter more than raw speed. Their pipeline also illustrates the tradeoff that drove the next generation: every extra stage adds accuracy and latency.

One-stage detectors: look once

YOLO, short for You Only Look Once, reframed detection as a single regression problem. Instead of proposing regions and then classifying them, a single network looks at the whole image and outputs boxes and classes directly.

The original design divides the image into a 7 by 7 grid. Each cell is responsible for objects whose centers fall inside it and predicts a small number of boxes with confidence scores, plus class probabilities. For the PASCAL VOC setup with two boxes per cell and 20 classes, that is 5 numbers per box times 2, plus 20, which makes 30 numbers per cell and 7 by 7 by 30, or 1,470 numbers, for the entire image in one forward pass.

Redmon and colleagues reported about 45 frames per second for the base model and about 155 for Fast YOLO. Their abstract is candid about the price: YOLO made more localization errors than state-of-the-art systems, but was far less likely to predict false detections on background (arXiv:1506.02640).

SSD and the focal loss insight

SSD, the Single Shot MultiBox Detector, predicted boxes from several feature maps of different resolution, so large and small objects were handled at different scales. Its authors reported 72.1 percent mAP on VOC2007 test at 58 frames per second with 300 by 300 input, and 75.1 percent with 500 by 500 input, better than a comparable Faster R-CNN (arXiv:1512.02325).

One-stage detectors still trailed two-stage ones in accuracy until RetinaNet, from Lin and colleagues, diagnosed why. The authors argued that "the extreme foreground-background class imbalance encountered during training of dense detectors" was the central cause. For illustration, a dense detector might evaluate on the order of a hundred thousand candidate locations in one image where only a handful contain objects; the easy empty locations drown out the rare informative ones. Their fix, focal loss, down-weights examples the model already classifies well so training focuses on the hard cases. RetinaNet reached accuracy that surpassed the two-stage detectors of its time at one-stage speed, according to its authors (arXiv:1708.02002).

A second analogy helps here. A two-stage detector is like a security team that first shortlists suspicious people in a crowd, then questions each one. A one-stage detector scans the whole crowd at once and flags matches in a single look. The first is more deliberate; the second is faster, and focal loss taught it to stop being distracted by the thousands of unremarkable faces.

Anchors, non-maximum suppression, and the move to anchor-free

Early one-stage and two-stage detectors relied on anchor boxes: a fixed set of template rectangles of different shapes and sizes placed at every location. The network learned to nudge each template toward a real object. Anchors work, but they add settings that must be tuned to the dataset, such as how many templates, what aspect ratios, and what overlap counts as a match.

FCOS (Fully Convolutional One-Stage detection) removed them, predicting objects pixel by pixel in the way semantic segmentation does and describing itself as "anchor box free, as well as proposal free." Its authors argue that dropping anchors eliminates hyperparameters that strongly affect performance (arXiv:1904.01355).

Non-maximum suppression

Dense detectors predict many overlapping boxes for the same object. Non-maximum suppression (NMS) cleans them up: keep the highest-confidence box, discard any other box that overlaps it beyond a threshold, and repeat. Think of a committee in which everyone submits the same recommendation and only the most confident vote is kept. NMS is effective, but it is hand-tuned, it can wrongly delete two real objects that overlap closely, and it adds latency after the network finishes. Several later designs aim to remove it.

YOLO since 2020

The YOLO name has been carried by several groups and is not a single lineage. Ultralytics released YOLOv8 on January 10, 2023, with what its documentation calls an anchor-free split head. Its published COCO results at 640-pixel input illustrate the speed and accuracy range within one family:

Model	COCO mAP (val, 50-95)	Latency on A100 with TensorRT (ms)	Approximate throughput
YOLOv8n	37.3	0.99	about 1,000 images per second
YOLOv8s	44.9	1.20	about 830 images per second
YOLOv8m	50.2	1.83	about 550 images per second
YOLOv8l	52.9	2.39	about 420 images per second
YOLOv8x	53.9	3.53	about 280 images per second

Source: Ultralytics YOLOv8 documentation. Throughput is the reciprocal of the listed latency, so it excludes data loading and other overheads in a real pipeline. Going from the smallest to the largest model buys 16.6 mAP points and costs more than three and a half times the latency.

Transformer detectors: DETR and its descendants

In 2020, Carion and colleagues at Facebook AI Research proposed DETR, short for Detection Transformer, treating detection as direct set prediction. DETR removes anchor generation and NMS. A transformer encoder-decoder (see transformers explained simply and how transformers pay attention to words) receives image features and a fixed set of learned object queries, and each query outputs one predicted box and class. During training, a bipartite matching loss assigns each real object to exactly one prediction, which pushes the model to produce one answer per object instead of many overlapping ones. On COCO, the authors report accuracy and runtime matching the well-established Faster R-CNN baseline (arXiv:2005.12872).

The convergence problem

The elegance came with a cost. DETR trained slowly, needing hundreds of passes over the training set, and handled small objects poorly because attention over a full high-resolution feature map is expensive. Deformable DETR addressed this by letting attention focus on a small set of sampling points near a reference location, and its authors report better results than DETR, especially on small objects, with about one-tenth the training epochs (arXiv:2010.04159).

DINO (DETR with Improved deNoising anchOr boxes) added contrastive denoising training and other changes. Its authors report 49.4 AP at 12 epochs and 51.3 AP at 24 epochs with a ResNet-50 backbone, and, after pretraining on Objects365 with a Swin-L backbone, 63.3 AP on COCO test-dev (arXiv:2203.03605).

Making it real time

RT-DETR, a real-time DETR variant, targets speed directly. Its abstract reports 53.1 percent AP at 108 frames per second for the ResNet-50 version and 54.3 percent AP at 74 frames per second for ResNet-101, measured on a T4 GPU, and says these outperform previously advanced YOLO detectors in both speed and accuracy. The authors argue that NMS hurts YOLO's speed and accuracy and that an end-to-end transformer avoids that step. They also report RT-DETR-R50 beating DINO-R50 by 2.2 percent AP while running about 21 times faster (arXiv:2304.08069).

The YOLO camp answered. YOLOv10, by Wang and colleagues, proposes "consistent dual assignments" so that training does not need NMS at inference. The authors report YOLOv10-S as 1.8 times faster than RT-DETR-R18 at similar COCO AP with 2.8 times fewer parameters and FLOPs, and YOLOv10-B with 46 percent less latency than YOLOv9-C at the same performance (arXiv:2405.14458).

Detector family	Core idea	Headline reported result	Needs NMS?
Faster R-CNN (2015)	Region Proposal Network plus classifier	About 5 fps, 300 proposals per image	Yes
YOLO v1 (2015)	Single-pass grid regression	About 45 fps, more localization errors	Yes
RetinaNet (2017)	One-stage plus focal loss	Beat two-stage detectors of its time	Yes
FCOS (2019)	Anchor-free, per-pixel prediction	44.7% AP with ResNeXt-64x4d-101	Yes
DETR (2020)	Transformer set prediction	Matches Faster R-CNN on COCO	No
RT-DETR (2023)	Real-time hybrid transformer	53.1% AP at 108 fps (R50, T4)	No
YOLOv10 (2024)	NMS-free dual assignment training	1.8x faster than RT-DETR-R18 (S model)	No

Every figure above is a claim from the cited paper under its own hardware and settings. A detector that is faster on a T4 can lose on a phone chip, and a published result at 640 pixels says little about performance at 1,280. Treat cross-paper comparisons as rough guides, not rankings.

Open-vocabulary detection and foundation models

Standard detectors are closed-vocabulary: they can only find the classes they were trained on, such as the 80 in COCO, or a few more than a thousand in the LVIS benchmark. A warehouse system trained on last year's products cannot find a new product without retraining.

Open-vocabulary detectors accept a text description. Grounding DINO combines the DINO detector with grounded pretraining and language-vision fusion modules (a feature enhancer, language-guided query selection, and a cross-modality decoder). Its authors report 52.5 AP on COCO in a zero-shot setting, meaning without any COCO training data, and a mean of 26.1 AP on the ODinW zero-shot benchmark (arXiv:2303.05499). Given a phrase such as "red backpack near the door," it can localize a match even if backpacks were never among its fixed labels.

Segmentation has a parallel story. Meta's Segment Anything Model (SAM) was trained on SA-1B, a dataset with more than a billion masks across 11 million licensed, privacy-respecting images, and is designed to be promptable, so a point or box can define what to segment (arXiv:2304.02643). SAM 2 extends the approach to video with a streaming memory; its authors report it is more accurate and six times faster than SAM on images, and needs three times fewer interactions on video (arXiv:2408.00714).

These models underpin a practical workflow: use a large open-vocabulary model to label images automatically, review the labels, then train a small, fast detector on the result. That pattern shifts effort from drawing boxes to checking them. It also connects to the broader idea of training smaller models from larger ones, covered in AI model distillation and, for generated training images, synthetic data.

Where detectors are used, and what accuracy means there

Driving

Perception for driver assistance and autonomy uses detectors on camera feeds, often fused with other sensors such as radar and LiDAR. The requirements differ from a benchmark: the system must handle night, rain, glare, and rare objects, track each object over time, and decide under uncertainty. A 0.5 IoU match that satisfies COCO may be too loose for judging whether a cyclist is inside the lane. For the wider system, see how self-driving cars work.

Medicine

Medical imaging is an instructive case because it shows the difference between detection and the nearby task of grading. In a 2016 JAMA study, Gulshan and colleagues trained a deep network on 128,175 retinal images, each graded by multiple ophthalmologists, and evaluated it for referable diabetic retinopathy on two datasets. The area under the ROC curve (AUC) was 0.991 on the EyePACS-1 set and 0.990 on Messidor-2. At a high-sensitivity operating point the sensitivity was 97.5 percent with 93.4 percent specificity on EyePACS-1 (JAMA, 2016).

That study is image-level classification, not box detection, but the lesson transfers: a headline AUC depends on an operating point, a dataset, and a population, and prevalence in the test set (about 7.8 percent referable cases in EyePACS-1) affects how useful a positive result is in practice. For a deeper look at detection models in cancer imaging, see AI cancer detection.

Retail and logistics

Cashierless stores and warehouse systems use detectors to track which items a person lifts, returns, or places in a bin. These settings are hard in ways benchmarks barely test: hands partly cover products, shelves are crowded, lighting changes, and the cost of a wrong label is a billing error or a mis-shipped order. A fixed product catalog makes closed-vocabulary detection workable, but every new product requires fresh training data or an open-vocabulary fallback.

Putting a detector on a small device

Accuracy on a data-center GPU does not transfer to a phone. Edge deployment relies on a toolkit of compression methods:

• Quantization: store weights as 8-bit integers instead of 32-bit floating-point numbers. That is a fourfold memory reduction by arithmetic, and often small accuracy loss when done carefully. See LLM quantization explained, which applies the same idea to language models.
• Pruning: remove weights or channels that contribute little.
• Distillation: train a small student model to imitate a large teacher.
• Architecture choice: pick a model family designed for the target chip, since convolutional layers are heavily optimized in mobile runtimes while some attention operations are not.

Developers also have to measure the whole pipeline, not just the network: camera capture, resizing, post-processing, and drawing results. A model that runs in 3.5 milliseconds on a server GPU may take far longer end to end on a battery-powered device. Edge AI on your phone covers how this works in practice.

What goes wrong with object detectors

Detectors have predictable weak spots, and knowing them helps interpret failures:

1. Small objects. A distant bicycle may span only a handful of pixels. COCO reports small-object AP separately because it is usually the lowest number.
2. Occlusion and crowding. Overlapping objects confuse both the network and NMS.
3. Domain shift. A model trained on daytime street photos can degrade on infrared, aerial, or underwater images.
4. Label noise. Annotators disagree about where a box ends, and a model trained on inconsistent boxes learns the inconsistency.
5. Class imbalance. Rare classes get few examples, and mAP, as an average, can hide a class that fails completely.
6. Benchmark overfitting. Repeated tuning against a fixed test set can inflate reported numbers; see why training and test data must be separated and overfitting and underfitting.

A single mAP score also averages away which mistakes matter. Two detectors with the same mAP can differ sharply on the cases a particular application cares about, which is why teams building real systems evaluate on their own data and look at per-class precision and recall, not just the headline figure. For broader context on benchmarks, see why AI benchmarks can be misleading.

The takeaway

Each generation of object detectors solved a bottleneck left by the previous one. Region proposals fixed the sliding window's cost, shared computation fixed R-CNN's redundancy, single-pass grids fixed the speed of two-stage pipelines, focal loss fixed the accuracy gap, anchor-free heads fixed the tuning burden, and transformers fixed the hand-built post-processing. Open-vocabulary models are now attacking the last major constraint: the fixed list of labels.

What stays constant is that detection is a measurement problem as much as a modeling one. A box is only useful if it is tight enough, fast enough, and right often enough for the decision it feeds, and a benchmark number is only as meaningful as its dataset, hardware, and threshold. Reading the setup behind the number is the skill that makes the technology legible.

FAQ

Q: What is mAP and how is it measured for object detection? A: Mean average precision summarizes detection accuracy across classes. A detection counts as correct when its class is right and its IoU with a real box passes a threshold. Average precision is the area under the precision-recall curve for one class, and mAP averages that over classes. COCO's headline metric additionally averages over IoU thresholds from 0.5 to 0.95.

Q: What is the difference between object detection and instance segmentation? A: Detection outputs a rectangle and a label for each object. Instance segmentation outputs a pixel-level mask for each object, which captures shape and separates touching objects. It costs more computation, and is used where outlines matter, such as medical tools, robotics grasping, and image editing. Models like SAM are designed specifically for promptable segmentation.

Q: Why are YOLO models often preferred on edge hardware? A: Convolutional networks have been heavily optimized in mobile and embedded runtimes, and the YOLO family publishes a range of model sizes, from very small to large. Transformer detectors such as RT-DETR report strong speed on server GPUs, but support and efficiency on small chips vary by device and software stack, so the answer is worth testing on the target hardware.

Q: How much training data does a custom object detector need? A: There is no universal number. Fine-tuning a pretrained detector needs far fewer examples than training from scratch, and the amount depends on how visually varied the classes are, how consistent the labels are, and how closely the deployment images match the training images. Label quality and coverage of hard cases usually matter more than raw image count.

Q: What does open-vocabulary detection mean? A: It means the detector accepts a text prompt instead of choosing from a fixed label list. Grounding DINO, for example, reports 52.5 AP on COCO without COCO training data. Open-vocabulary models are more flexible but are often larger and slower than a small closed-vocabulary model trained for one job.

Q: How are detectors made small enough for phones and cameras? A: Through quantization, pruning, distillation, and choosing architectures suited to the target chip. Quantizing 32-bit weights to 8 bits cuts memory by a factor of four. Developers should also measure end-to-end latency on the actual device, since preprocessing and post-processing can dominate the network's own run time.

Q: Why do object detectors miss small or crowded objects? A: Small objects occupy few pixels and lose detail as images are downsampled through the network, while crowds create overlapping boxes that NMS may merge or suppress. This is why COCO reports small-object AP separately, and why newer designs focus on multi-scale features and NMS-free training.

Sources

• Girshick et al., R-CNN: arxiv.org/abs/1311.2524
• Girshick, Fast R-CNN: arxiv.org/abs/1504.08083
• Ren et al., Faster R-CNN: arxiv.org/abs/1506.01497
• Redmon et al., YOLO: arxiv.org/abs/1506.02640
• Liu et al., SSD: arxiv.org/abs/1512.02325
• Lin et al., RetinaNet and focal loss: arxiv.org/abs/1708.02002
• Tian et al., FCOS: arxiv.org/abs/1904.01355
• Carion et al., DETR: arxiv.org/abs/2005.12872
• Zhu et al., Deformable DETR: arxiv.org/abs/2010.04159
• Zhang et al., DINO: arxiv.org/abs/2203.03605
• Zhao et al., RT-DETR: arxiv.org/abs/2304.08069
• Wang et al., YOLOv10: arxiv.org/abs/2405.14458
• Liu et al., Grounding DINO: arxiv.org/abs/2303.05499
• Kirillov et al., Segment Anything: arxiv.org/abs/2304.02643
• Ravi et al., SAM 2: arxiv.org/abs/2408.00714
• Lin et al., Microsoft COCO: arxiv.org/abs/1405.0312
• Ultralytics YOLOv8 and COCO documentation: docs.ultralytics.com
• Gulshan et al., JAMA 2016: jamanetwork.com

---

Read next:

• The Transformer Architecture Explained: Why Attention Changed Everything
• How Do Self-Driving Cars Actually Work?
• AI-Powered Software Testing: How Machine Learning Finds Bugs Before You Do