How AI Detects Patterns In Medical Scans

A radiologist reading a chest X-ray draws on years of training and thousands of prior cases. An AI model looking at the same image relies on something structurally similar but mechanically different: millions of numerical filters tuned by exposure to labeled images, with no built-in concept of what a lung, a tumor, or a fracture actually is. That distinction explains both what these systems do well and where they fail. This article covers how the mechanism works, what the major studies found, which tools regulators have cleared, and the documented limits that keep most of them in a supporting role.

What a convolutional neural network does with pixels

Most medical imaging AI is built on a convolutional neural network (CNN), an architecture that scans an image with small filters. A filter is a tiny grid of numbers, perhaps three pixels by three, that slides across the image and measures how strongly each patch matches a pattern such as an edge, a texture, or a boundary between bright and dark tissue. The result is a new image, called a feature map, that lights up wherever the pattern appears.

The first layer of a CNN learns simple filters. The next layer combines those outputs into slightly larger patterns, such as a curve or a cluster of bright dots. After dozens of layers the network responds to complex arrangements: the rounded density of a nodule, the branching of a vessel, the speckled texture of a certain tissue. Pooling steps shrink the maps along the way, so deeper layers see a wider region of the original image at once.

None of this happens because the model understands anatomy. During training, the network's parameters (often millions of them) are nudged, image by image, so that its output moves closer to the label a human expert assigned. This is supervised learning: if a pixel pattern appeared alongside the label "cancer" often enough in the training set, the weights shift to detect it. It is a curve-fitting exercise over expert-provided labels. For a wider look at the same ideas, see our explainer on how neural networks work.

An apprentice who has never taken anatomy

Picture an extremely persistent apprentice shown 100,000 flagged X-rays and 100,000 unflagged ones. Nobody explains what a lung is. Through repetition alone, the apprentice learns which pixel arrangements tend to land in the flagged pile. That is powerful, because the apprentice never tires and applies the same standard to image 100,000 as to image one. It is also brittle, because the apprentice has no way to tell when a pattern is a symptom and when it is just a quirk of how the pictures were taken.

What a model actually outputs

The raw output depends on the task. A classification model returns a score, such as a probability that an image contains referable disease. A detection model returns boxes around suspicious regions. A segmentation model labels every pixel, outlining an organ or a tumor. The U-Net architecture, introduced in 2015 by Olaf Ronneberger, Philipp Fischer, and Thomas Brox, became a standard for segmentation because it could be trained "from very few images" and segment a 512 by 512 image in under a second on a GPU of that era (Ronneberger et al., arXiv, 2015).

Transformers enter the picture

CNNs are no longer the only option. In 2020, Alexey Dosovitskiy and colleagues showed that a standard transformer, the architecture behind large language models, could be applied to sequences of image patches and match leading CNNs when pre-trained on large amounts of data (Dosovitskiy et al., arXiv, 2020). Medical groups have adapted these vision transformers, and they handle global context differently from convolutions. If you are curious about the underlying mechanism, our piece on how transformers pay attention to words explains attention in the language setting, where it started.

Where the training labels come from

A model is only as good as the labels it learns from. Consider diabetic retinopathy, damage to the retina's blood vessels caused by long-term high blood sugar. In a landmark 2016 study in JAMA, Varun Gulshan and colleagues at Google trained a CNN on 128,175 retinal images, each graded between three and seven times by a pool of 54 ophthalmologists and senior residents. The network trained on 80 percent of the images and was tuned on the remaining 20 percent (Gulshan et al., JAMA, 2016).

The team then validated the model on two separate sets: EyePACS-1 with 9,963 images from 4,997 patients, and Messidor-2 with 1,748 images from 874 patients. The model's operating point was adjustable, and that matters for the table below.

Operating point	Dataset	Sensitivity	Specificity
High specificity	EyePACS-1	90.3%	98.1%
High specificity	Messidor-2	87.0%	98.5%
High sensitivity	EyePACS-1	97.5%	93.4%
High sensitivity	Messidor-2	96.1%	93.9%

Source: Gulshan et al., JAMA, 2016, for referable diabetic retinopathy. Area under the curve was 0.991 on EyePACS-1 and 0.990 on Messidor-2.

The study shows a basic fact about how these models are built: the accuracy ceiling is tied to the quality and consistency of the labels. The model reproduces the pattern in the labeled data. It does not measure an independent biological truth about the patient's retina. When expert graders disagree on borderline cases, which happens even among specialists, that disagreement gets baked into what the model learns to call "disease."

Chest X-rays and the CheXNet comparison

A similar pattern appears for chest films. The Stanford ML Group's CheXNet is a 121-layer CNN trained on ChestX-ray14, described at the time as the largest public chest X-ray dataset, with more than 100,000 frontal images covering 14 diseases. In a 2017 paper, Pranav Rajpurkar and colleagues reported that on pneumonia detection the model's F1 score (a measure combining precision and recall) exceeded the average of four practicing academic radiologists who annotated the test set (Rajpurkar et al., arXiv, 2017). The comparison also hints at something less often discussed: how much interpretation varies among trained humans, not only between humans and models.

In 2018 the FDA authorized IDx-DR, an autonomous AI system that detects diabetic retinopathy without a specialist reading the image. Its pivotal trial enrolled 900 adults with diabetes at primary care clinics and reported 87.2% sensitivity, 90.7% specificity, and 96.1% imageability, meaning the camera and software produced a usable result in most patients without a specialist grading the photograph ([Abramoff et al., npj Digital Medicine, 2018](https://www.ebi.ac.uk/europepmc/webservices/rest/search?query=%22Pivotal%20trial%20of%20an%20autonomous%20AI-based%20diagnostic%20system%22%20diabetic%20retinopathy&resultType=core&format=json)).

Four more bodies of evidence, in short

Retinal photos and chest films are only part of the literature. The studies below involve other imaging types and help show what "matching experts" means in practice.

Study	Modality and task	What was reported
McKinney et al., Nature, 2020	Screening mammography, UK and US data	Versus human readers, false positives fell 5.7% (US) and 1.2% (UK); false negatives fell 9.4% (US) and 2.7% (UK). In a simulated UK double-reading workflow, the second reader's workload dropped 88%
Ardila et al., Nature Medicine, 2019	Low-dose chest CT, lung cancer risk	AUC of 94.4% on 6,716 National Lung Screening Trial cases; without prior scans the model beat six radiologists, with 11% fewer false positives and 5% fewer false negatives (absolute)
Pantanowitz et al., Lancet Digital Health, 2020	Prostate biopsy slides, cancer detection	AUC 0.997 internally and 0.991 on 100 consecutive cases scanned on a different scanner; used as a second read on 11,429 slides
Raciti et al., Modern Pathology, 2020	Prostate biopsy slides, reader study	Three pathologists reviewing 304 slides: average sensitivity 74% unaided and 90% with the Paige Prostate Alpha system, with no significant change in specificity

Sources: McKinney et al., Ardila et al., Pantanowitz et al., Raciti et al. (all via Europe PMC records).

Read these numbers with their design in mind. The McKinney work compared the model with the original readers' decisions on retrospective data and, in a separate study, with six radiologists; the UK set was representative while the US set was enriched with cancers. The Ardila work compared the model with six radiologists in reader studies rather than in routine practice. Retrospective comparisons are valuable but are not the same as a prospective trial in daily practice.

From retrospective studies to a real screening trial

One of the more useful tests of whether the lab results carry over to real workflows is the MASAI trial in Sweden. It randomized 80,033 women aged 40 to 80 at four screening sites between April 2021 and July 2022: 40,003 to AI-supported reading and 40,030 to standard double reading by two radiologists (MASAI trial safety analysis, The Lancet Oncology, 2023).

The reported results of this safety analysis:

• Cancer detection rate: 6.1 per 1,000 with AI support versus 5.1 per 1,000 in the control group.
• Recall rate (women called back for further tests): 2.2% versus 2.0%.
• False positive rate: 1.5% in both groups.
• Screen-reading workload: reduced by 44.3% in the AI-supported group.

The authors judged the AI-supported approach safe enough to continue. The primary endpoint, interval cancers (cancers that appear between scheduled screens), was still pending in that report. That gap is the right way to hold the result: a workload reduction and similar recall are encouraging, but they do not yet answer whether fewer cancers are missed over time.

Detection-assistance versus autonomous diagnosis

Not every claim that "AI reads your scan" means the same thing, and the difference has clinical and legal weight. Most imaging AI on the market is cleared as detection assistance, sometimes called computer-aided detection or diagnosis (CAD or CADx). The software flags a region or produces a risk score. A licensed clinician reviews it and makes the final call. The AI is a second set of eyes.

Autonomous diagnostic AI is a narrower and more tightly regulated category in which the software's output can be acted on without a specialist independently reviewing the image. IDx-DR belongs here, but only for one task: detecting more-than-mild diabetic retinopathy or diabetic macular edema in adults with diabetes who have no prior diagnosis of retinopathy, in primary care settings using a specified camera and workflow. The trial population was a median age of 59, and 23.8 percent of participants had the target condition (Abramoff et al., 2018). Regulators evaluate such systems against the exact task, image type, and population tested, not against a broad claim.

A comparison from digital pathology: the FDA granted a De Novo decision to Paige Prostate on September 21, 2021, under a classification described as "software algorithm device to assist users in digital pathology" (FDA De Novo database, DEN200080). The classification name itself signals the intent: assistance, with a pathologist still responsible for every case.

Why the distinction matters

If a device is cleared for assistance, responsibility for the diagnosis sits with the clinician, and the output works more like a spell-checker than a verdict. If a device is cleared as autonomous, its decision can be acted on within the validated task, but the authorization does not transfer to any other use of the same underlying model.

The FDA maintains a public list of AI-enabled medical devices authorized for marketing in the United States. It selects entries mainly by searching authorization documents for AI-related terms and cautions that the list "is not a comprehensive resource of AI-enabled medical devices" (FDA list of AI-enabled medical devices). Radiology is the most common category among the entries visible on the page, but the page itself gives no exact share, so this article does not quote one.

Why models miss things a trained human would not

CNNs detect statistical correlations between pixel patterns and labels, and that creates a documented failure mode: the model latches onto shortcuts that have nothing to do with disease. A network might associate a scanner's markings, a positioning habit, or a compression artifact with a diagnosis because that correlation existed in the training set. The research literature calls this shortcut learning.

The Mount Sinai pneumonia study

A clear demonstration came from John Zech and colleagues in PLOS Medicine in 2018. They tested pneumonia-detecting CNNs on 158,323 frontal radiographs from three sources: the NIH Clinical Center, Mount Sinai Hospital, and Indiana University. Internal performance exceeded external performance in three of five comparisons. For example, a model trained on Mount Sinai data scored an internal AUC of 0.802 but 0.717 when tested at NIH (Zech et al., PLOS Medicine, 2018).

The reasons are instructive. Pneumonia prevalence differed sharply between sites: 34.2 percent at Mount Sinai, 1.2 percent at NIH, and 1.0 percent at Indiana. A model that ignored the images and ranked patients only by their hospital's prevalence reached an AUC of 0.861 on the combined Mount Sinai and NIH test set. The CNNs could also identify the source hospital in over 99 percent of NIH and Mount Sinai radiographs, and they separated Mount Sinai inpatient-ward films from emergency-department films with 100 percent accuracy. The authors concluded that internal test results can overstate performance at new sites and that external validation is essential.

A wine-taster analogy

Imagine training a wine taster exclusively on bottles from a single vineyard, then asking them to judge wine from a vineyard across the country with different grapes, soil, and climate. Some knowledge transfers. Some habits specific to the first vineyard actively mislead in the second. A CNN trained on one hospital's imaging pipeline faces the same problem, except it usually cannot tell you when it has left familiar territory. Our article on why training and test data must be separated covers the statistical side, and why data leakage breaks AI evaluations explains how a flattering test result can come from the wrong source.

Can a model trained at one hospital work reliably at another?

Not automatically. Published work such as Zech et al. shows measurable performance drops when models meet scanners, protocols, or patient populations that differ from training. A preprint by DeGrave and colleagues, on COVID-19 detection from chest radiographs, reported that such systems relied on confounding factors rather than medical pathology and failed when tested at new hospitals (DeGrave et al., medRxiv, 2020). Pantanowitz and colleagues addressed the issue in their prostate work by testing on cases scanned with a different scanner, and the drop was small (AUC 0.997 internal versus 0.991 external), which is why such external tests are now a standard expectation. A model also drifts after deployment as equipment and patient mix change, a topic covered in what model drift means in production.

Seeing what the model looked at

Clinicians reasonably ask why a model flagged a scan. Visual explanation methods try to answer. Grad-CAM, published in 2016, produces a "coarse localization map highlighting important regions in the image" by using the gradients that flow into the network's last convolutional layer (Selvaraju et al., arXiv). The result is the colored heatmap often shown over an X-ray.

These maps are helpful, with a caveat. A heatmap shows where the network's response was strong. It does not prove that the highlighted pattern is clinically meaningful. In the shortcut examples above, a heatmap could land on a text marker or the image edge. Researchers working on deeper approaches, covered in our article on mechanistic interpretability, are trying to trace what individual circuits compute, but this remains an active research area, not a solved one.

The sensitivity and specificity tradeoff

Every imaging model is tuned between two competing priorities. Sensitivity is the share of people with the disease whom the test correctly flags. Specificity is the share of people without the disease whom the test correctly clears. Pushing toward sensitivity, often the priority for screening because a missed disease is the costlier error, generally raises false positives, meaning more healthy people are called back.

The Gulshan table shows this directly: moving from the high-specificity to the high-sensitivity operating point on EyePACS-1 raised sensitivity from 90.3 to 97.5 percent while specificity fell from 98.1 to 93.4 percent. The same model, a different threshold. The tradeoff is formalized in the receiver operating characteristic (ROC) curve, which plots sensitivity against the false-positive rate at every possible threshold. Different vendors pick different points for different intended uses.

Why prevalence changes what a positive result means

The same sensitivity and specificity can mean very different things in different populations. Here is an illustration using the IDx-DR trial figures of 87.2 percent sensitivity and 90.7 percent specificity. These numbers are applied to hypothetical populations, so treat the results as arithmetic, not as measured outcomes.

Share of screened people with the disease	Out of 1,000 screened: true positives	False positives	Share of positive results that are correct
23.8% (the trial's rate)	about 208	about 71	about 75%
5%	about 44	about 88	about 33%
1%	about 9	about 92	about 9%

When a condition is rare among the people screened, most positives are false even for a strong test. This is a property of probability, not of the model, and it is why a tool validated in one setting needs a fresh look in another. Our explainer on precision, recall, and class imbalance walks through the same logic for machine learning generally.

Where AI-assisted imaging is used today

Adoption varies by modality, task, and the amount of clinical responsibility the software is allowed to carry. The table summarizes where CNN-based tools have moved beyond research, based on the studies above and the regulatory records cited.

Modality	Example task	What the AI flags	Evidence cited in this article	Role
Retinal photography	Diabetic retinopathy screening	Microaneurysms, hemorrhages, vascular lesions	Gulshan 2016; IDx-DR pivotal trial 2018	Autonomous for one narrow task
Chest X-ray	Pneumonia and other findings	Regions of abnormal density	Rajpurkar 2017; Zech 2018	Research and assistive tools
Chest CT	Lung cancer risk	Nodules and change over time	Ardila 2019	Research and assistive tools
Digital pathology	Prostate cancer in biopsies	Cellular patterns tied to malignancy	Raciti 2020; Pantanowitz 2020; FDA De Novo 2021	Assistive; pathologist signs off
Mammography	Breast cancer screening	Masses, calcifications, distortion	McKinney 2020; MASAI trial 2023	Studied as reader support

Detection-assistance tools that keep a clinician in the loop dominate real-world deployment. Autonomous clearance remains rare and narrowly scoped.

Cleared status of specific commercial products in mammography and CT varies and changes often, so this article does not list vendor names. The FDA list linked above is the place to check any particular product.

What people get wrong about "AI reads scans now"

"An FDA clearance means the AI diagnoses the patient." Most cleared imaging AI is assistive software that a licensed clinician reviews. The one autonomous example discussed here, IDx-DR, is scoped to a single screening task in a defined population.

"A published accuracy number applies everywhere." A validated sensitivity and specificity belong to the dataset, the scanner, and the population tested. Zech et al. found that models identified hospital sources with near-perfect accuracy, which is a reminder of how much non-disease signal an image carries.

"If it beat the average radiologist, the task is solved." Matching expert performance on a defined benchmark is a real result. It differs from replacing the judgment of a clinician who weighs a scan against a patient's history, symptoms, and other tests, which are context most imaging models do not receive. The CheXNet comparison used four radiologists; the McKinney study used six. Small reader panels give wide uncertainty about how any individual clinician would fare.

"AI makes human readers less necessary in screening." The MASAI trial is the closest evidence to date. It found a lower reading workload and similar recall in the AI-supported arm, with the interval cancer result still pending. That is evidence about efficiency and safety so far, not a verdict on replacing readers.

"The model understands the disease." A model also has no way to recognize that an image is outside its training distribution. If it mislabels something unfamiliar, it will usually do so with confidence. For a related issue in language models, see why AI hallucinates facts.

How the field checks itself

Several practices have become standard in credible studies. They are worth knowing when you read a headline about a new model.

1. External validation. The model is tested on data from hospitals, scanners, or time periods not used in training. The Pantanowitz study and the Zech study both illustrate why.
2. Reader studies. Several clinicians read the same cases with and without the AI, so the study measures the human-plus-AI pair as well as the model.
3. Prospective trials. The MASAI trial randomized real screening participants rather than replaying old images.
4. Pre-specified endpoints. The IDx-DR trial set superiority thresholds of 85 percent sensitivity and 82.5 percent specificity before the results came in.
5. Post-deployment monitoring. Performance is tracked after launch, because patient mix and equipment shift.

Feedback loops matter in training too; our article on how AI models learn from feedback explains the broader family of methods that adjust a model using human judgments. For the clinical side of the same story, see AI in healthcare and the cancer-specific look at AI cancer detection with deep learning. The imaging itself is worth understanding too: how MRI works describes one of the modalities these models now read.

FAQ

Q: Does AI understand what it sees in a medical scan the way a doctor does? A: No. A convolutional neural network detects statistical pixel patterns correlated with labels it saw during training. It has no independent concept of anatomy or disease progression, and it does not use a patient's history or symptoms unless that data is explicitly fed into the model.

Q: How accurate is AI at detecting disease in medical images? A: It depends on the task, dataset, and setting. Gulshan et al. reported 90.3 percent sensitivity and 98.1 percent specificity at one operating point on EyePACS-1, and the IDx-DR trial reported 87.2 and 90.7 percent. These are results for specific validation sets and can shift at a different hospital.

Q: Can a model trained at one hospital work reliably at another? A: Not automatically. Zech et al. found internal performance exceeded external performance in three of five comparisons, and models could identify the source hospital in over 99 percent of some test sets. External validation on outside data is the standard check.

Q: Has the FDA authorized AI to diagnose patients without a doctor? A: In one narrow, well-documented case, yes. IDx-DR was authorized in 2018 as an autonomous system for diabetic retinopathy screening in primary care, and its authorization does not cover other uses. Most other cleared imaging AI, including Paige Prostate in pathology, is assistive.

Q: Is AI going to replace radiologists or pathologists? A: The evidence in this article does not show that. The large prospective trial so far, MASAI, reported reduced reading workload with similar recall, and its main endpoint was still pending. The cleared tools discussed here keep a clinician responsible for the final decision.

Q: Why does a heatmap on a scan not prove the AI is right? A: A heatmap such as Grad-CAM shows where the network's response was strong. It does not prove that region is clinically meaningful, and models can attend to irrelevant markers. It is a debugging aid, not a diagnosis.

The takeaway

Medical scan AI works by learning statistical pixel patterns tied to expert labels, at a scale and consistency no individual reader matches across a large dataset. It has no understanding of anatomy, disease progression, or patient context. That is why the strongest results, from Google's retinopathy model to the McKinney mammography study and the MASAI trial, describe narrowly validated tasks, and why the hospital-shift findings of Zech and colleagues are the right counterweight to a headline accuracy number. The practical question about any scan-reading tool is not "how smart is it" but "on which images, from which scanners, in which population was it checked, and who is responsible for the final call."

This article is educational and does not provide medical advice. Talk to a qualified clinician about any imaging result or screening decision.

---

Read next:

• How transformers pay attention to words
• Why AI hallucinates facts
• How AI models learn from feedback

Sources

• Gulshan V, et al. JAMA, 2016
• Rajpurkar P, et al. CheXNet, arXiv, 2017
• Zech JR, et al. PLOS Medicine, 2018
• FDA De Novo database entry for Paige Prostate and AI-enabled device list
• Ronneberger O, et al. U-Net, arXiv, 2015; Dosovitskiy A, et al. Vision Transformer, arXiv, 2020; Selvaraju R, et al. Grad-CAM, arXiv