Neural Networks and the Microscope: How AI Sees Biology in New Ways

A modern drug screen can photograph millions of cells in a week, and no person can look at them all. Neural networks now outline each cell, clean up images taken with very little light, predict fluorescent labels without staining, and turn each cell's appearance into numbers that can be compared across thousands of experiments. This article explains how those tools work, which are well tested, and why a model that excels on its training data can still fail on a new kind of cell.

Key takeaways

Segmentation, deciding which pixels belong to which cell, is the step everything else depends on, and neural networks now handle it better than hand-built algorithms on many image types.
Restoration networks such as CARE can recover usable images from far less light, which matters because light damages living cells.
Cell Painting turns cell appearance into profiles of about 1,500 measurements, and large public datasets now exist for training and testing models.
Benchmarks from the Cell Tracking Challenge show deep learning leads on segmentation but generalizes poorly to data unlike its training sets.

Why microscopy became a data problem

For most of its 350-year history, microscopy was an act of looking. A researcher examined a slide, drew or photographed what appeared, and described it in words. The story of that tradition, from Leeuwenhoek and Hooke to Hooke's drawing of cork cells, is one of human eyes finding structure. Automated microscopes changed the scale. A robotic system can image a 384-well plate, capture several fields per well, and repeat the process hundreds of times in a campaign.

The numbers get large quickly. In 2020, Recursion Pharmaceuticals said its database held more than 5 petabytes of biological images generated in its own laboratories (Recursion). The company's fourth-quarter 2024 update said it ran up to 2.2 million experiments per week. The JUMP Cell Painting dataset, a public resource, was described in a 2023 preprint as roughly 115 terabytes covering about 1.6 billion cells. A person can examine a few hundred images carefully. Nobody can examine a billion cells, so a model must.

Pixels in, measurements out

A microscope image is a grid of numbers: each pixel records how much light reached the detector. Turning that grid into biology requires three steps. First, find the cells and the structures inside them. Second, measure each one for size, shape, brightness, and texture. Third, compare the measurements across conditions, such as treated versus untreated. Neural networks help at each step, and the first step is where they made the largest difference.

The models used are mostly convolutional neural networks, which scan an image with small filters that respond to edges, blobs, and textures, then combine those responses into higher-level features. The general mechanism is explained in how image recognition works and computer vision AI. Biology adds twists: the images often have more than three color channels, may be three-dimensional, and come with far fewer labeled examples than photographs of everyday objects.

Teaching a network to find cells

Segmentation means assigning every pixel to a cell or to background, and often separating touching cells from one another. It sounds simple. In practice, cells touch, overlap, vary wildly in shape, and look different under different stains and microscopes. Classical methods such as thresholding (call everything brighter than a cutoff a cell) and watershed algorithms (flood the image from seed points until the floods meet) work when cells are well separated and evenly stained, and break down when they are not.

The U-Net

The turning point was the U-Net, introduced by Olaf Ronneberger, Philipp Fischer, and Thomas Brox at MICCAI 2015. It uses a contracting path to capture context and a symmetric expanding path to recover precise locations, and it can be trained from relatively few annotated images. On the ISBI 2015 cell tracking challenge, the authors took first place in the phase-contrast and DIC categories by a wide margin, and they reported segmenting a 512 by 512 image in under a second on a recent graphics card (arXiv 1505.04597). Most of the tools below are U-Net descendants or relatives.

Cellpose

Cellpose, from Carsen Stringer, Marius Pachitariu, and colleagues, was published in Nature Methods in January 2021 as a "generalist" algorithm. Instead of predicting cell outlines directly, it predicts for each pixel a vector pointing toward the center of the cell it belongs to, and then follows those vectors to group pixels into cells. Picture a landscape where every cell is a basin: drop a marble on any pixel and it rolls toward that cell's center, and the marbles that end up in the same basin form one cell. The vector approach handles touching cells and irregular shapes better than boundary-finding, because it asks "which cell center do you belong to" rather than "where is the edge."

The training data matter as much as the architecture. The authors built a new, highly varied dataset of more than 70,000 segmented objects, including non-cellular images of repeated objects such as fruit and jellyfish, which they expected to improve generalization. The model needs no retraining or parameter tuning for many image types, and a three-dimensional version reuses the 2D model without needing 3D labels (PubMed record). Later versions added human-in-the-loop training so that users could improve the model on their own data (Cellpose 2.0, 2022) and one-click image restoration (Cellpose3, 2025). A 2025 preprint described Cellpose-SAM, which combines the Cellpose framework with the pretrained image encoder from Meta's Segment Anything Model. One licensing note: the models are trained on data licensed CC-BY-NC, which limits commercial use.

StarDist and micro-SAM

StarDist, by Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene Myers, takes a different view. It represents each cell nucleus as a star-convex polygon (a shape where every boundary point can be seen from the center) and predicts, for each pixel, the distances to the edge along a fan of rays. The idea suits round nuclei in crowded fluorescence images, and it was later extended to three dimensions as StarDist-3D (arXiv 1806.03535).

The newest family adapts foundation models. micro-SAM (μSAM), described by Anwai Archit and colleagues in Nature Methods in March 2025, fine-tunes Segment Anything for light and electron microscopy. According to a University of Göttingen release, the model was retrained on more than 17,000 microscopy images with over 2 million manually annotated structures, and it runs as a plugin for napari, an open-source image viewer, supporting both interactive and automatic segmentation.

Tool	Core idea	Best suited to	Published
U-Net	Encoder-decoder network that works with few labeled images	General biomedical segmentation, a base for many tools	MICCAI, 2015
StarDist	Predicts star-convex polygons per pixel	Round nuclei in fluorescence images	arXiv 2018; later 3D version
Cellpose	Predicts vectors toward each cell center	Many cell types without retraining	Nature Methods, 2021
micro-SAM	Segment Anything fine-tuned on microscopy data	Interactive and automatic segmentation, including electron microscopy	Nature Methods, 2025

Seeing more with less light

Fluorescence microscopy lights up molecules of interest with laser or lamp illumination. The trouble is that light is not harmless. Intense or prolonged illumination can bleach the fluorescent labels and damage or alter living cells, a problem called phototoxicity. Researchers watching a live process face a bargain: more light gives a cleaner picture and a sicker cell. Background on the physics is in how fluorescence works.

Content-aware restoration

Content-aware image restoration, or CARE, from Martin Weigert, Loic Royer, Florian Jug, and colleagues, appeared in Nature Methods in December 2018. A U-Net-style network learns to turn a noisy, low-light image into the clean version that a longer exposure would have produced. It trains on pairs: the same specimen imaged at low and high light. The paper demonstrated restoring images acquired with 60 times fewer photons, recovering near-isotropic resolution from up to tenfold axial undersampling, and resolving sub-diffraction structures at 20 times the frame rate of earlier methods. The authors released the tools as open source in Python, Fiji, and KNIME (PubMed record).

Think of an art restorer who has studied thousands of paintings in both damaged and pristine states. Shown a new damaged painting, the restorer can fill in what the original probably looked like, but the result is an informed reconstruction and not a recovery of lost paint. That distinction is the central caution for restoration networks.

What restoration can and cannot promise

Restoration networks learn from examples, so they are strongest when new images resemble the training data. They can sharpen structures they have seen, and they can also produce confident, clean-looking artifacts when the input differs from what they learned. For measurement, that matters: a restored image should be validated against the real specimen or against a held-out high-quality image before anyone quantifies it. Cellpose3's one-click restoration illustrates the practical use, which is improving segmentation of noisy images rather than creating publication art.

Predicting fluorescence without the dye

If a network can learn what a cell's nucleus looks like in a fluorescence image, can it find the nucleus in an unlabeled brightfield image? Researchers at the Allen Institute for Cell Science in Seattle showed that, for many structures, it can. Chawin Ounkomol, Gregory Johnson, and colleagues published a label-free method in Nature Methods in 2018 that predicts three-dimensional fluorescence images from transmitted-light images using a U-Net. They trained it on registered pairs of images: 30 pairs per structure for the 3D models, mainly from genome-edited human stem cells (PMC record).

The benefits are practical. Label-free imaging leaves cells undisturbed, avoids phototoxicity from fluorescence, and lets one transmitted-light recording yield several predicted structures at once, including time-lapse sequences. The limits are equally instructive. The method only works when a relationship exists between the transmitted-light signal and the fluorescence, and the authors reported that desmosomes and actomyosin bundles performed poorly. Models trained on one cell type performed worse on drastically different morphologies, accuracy fell when imaging parameters differed from training, and predicted intensity may not map directly onto protein amount, which rules out some measurements.

Cell Painting: giving cells a fingerprint

Segmentation tells you where the cells are. Cell Painting tells you what state they are in. The assay, developed at the Broad Institute and described in a 2016 Nature Protocols paper by Mark-Anthony Bray and colleagues, uses six fluorescent dyes imaged in five channels to label eight cellular components. Automated analysis then measures about 1,500 morphological features per cell, including size, shape, texture, and intensity, which together form a profile. Earlier profiling assays typically used only a few dyes.

Cellular component	Dye
Nucleus	Hoechst 33342
Endoplasmic reticulum	Concanavalin A
Nucleoli and cytoplasmic RNA	SYTO 14
Filamentous actin	Phalloidin
Golgi apparatus and plasma membrane	Wheat germ agglutinin (WGA)
Mitochondria	MitoTracker Deep Red

Phalloidin and WGA share one channel, which is how six dyes fit into five channels. The protocol was demonstrated in 13 cell cultures, most often the U2OS and A549 lines, with about 2,000 cells seeded per well in 384-well plates and nine fields imaged per well at 20 times magnification. One full cycle of culture and imaging takes about two weeks, and analysis adds another one to two weeks (PMC record).

Why profiles are useful

The premise of morphological profiling is that treatments with similar biological effects tend to produce similar cell appearances, even when the treatments are chemically unrelated. If so, a profile functions like a fingerprint: compare the fingerprint of an uncharacterized compound with a library of known ones, and a close match hints at what the compound does. Researchers use this to group compounds, flag likely toxicity, and connect genetic changes to cell states. It is a screening tool that generates hypotheses, not proof of a mechanism, and follow-up experiments are still needed.

Neural networks enter in two ways. They can segment the cells, and they can replace hand-designed features with learned ones, extracting a vector of numbers that describes each cell or well directly from the pixels. For the broader pattern of machine learning applied to cellular data, see single-cell AI, single-cell RNA sequencing, and machine learning in drug discovery.

Public data and industrial scale

The JUMP Cell Painting Consortium, which includes industry and academic partners, released a dataset that its 2023 preprint described as covering more than 116,750 chemical compounds, overexpression of 12,602 genes, and DNA sequences in living cells with letter-level accuracy.">CRISPR knockouts of 7,975 genes, imaged in U2OS cells. It is hosted in the Cell Painting Gallery on the AWS Registry of Open Data. The release is useful because it lets any group train and test models on a shared, large benchmark instead of private data. The project pages note that quality control was still being completed for parts of the data, so analyses should follow the current release notes.

Recursion Pharmaceuticals runs a commercial version of the idea. Its public RxRx3 dataset contains about 2.2 million images covering the human genome and 1,600 commercially available compounds in a single cell type, which the company said was under 1 percent of its total data. The company's claims about drug candidates are a separate matter from the imaging technology, and clinical results are the only test of those claims.

Tracking cells through time

A snapshot shows structure. A movie shows behavior: cells divide, migrate, die, and change shape. Tracking each cell across frames means segmenting every frame and then linking detections over time, which is hard when cells split into two, move fast, or crowd together. A cell division is one of the processes tracked, described in how cells divide, and so is programmed cell death, covered in how apoptosis works.

The field's reference benchmark is the Cell Tracking Challenge, launched in 2013 under the IEEE International Symposium on Biomedical Imaging. A 2023 update in Nature Methods, summarizing 10 years of results, offers a sober picture of where deep learning stands (PMC record):

The repository grew from 13 datasets in 2017 to 20.
Machine learning, mostly deep learning, dominates segmentation: 60 of the 89 submitted algorithms used it.
Four of the five top algorithms on the segmentation and tracking leaderboards used deep-learning-based segmentation, and U-Net variants were the most common architecture among top performers.
For linking cells across frames, the paper reported no notable performance difference between machine-learning and traditional methods.
Performance decreased significantly when pretrained methods were applied to different morphologies or imaging modalities.

That final point deserves emphasis. A network is a specialist trained on a particular visual diet. The benchmark's own authors found that pretrained methods did poorly on unfamiliar data types, which is why generalist models such as Cellpose, built on deliberately diverse data, attracted so much interest.

Where the models fail

Deep learning has not made microscopy analysis routine, and the failures tend to follow a few patterns.

Distribution shift. A model trained on one microscope, stain, and cell type can degrade silently on another. The same visual feature looks different under a different objective lens or a different camera. Readers who want the machine-learning side of this problem can see transfer learning and overfitting and underfitting.

Ground truth is expensive and subjective. Segmentation labels are drawn by people, and experts disagree about where a cell ends. Annotation takes hours per image, which is why labeled datasets stay small. Training and test labels drawn with different conventions can make a good model look bad, or the reverse.

Evaluation leaks. If images from the same experiment appear in both training and test sets, scores are inflated, as explained in why data leakage breaks AI evaluations. Good practice holds out entire plates, experiments, or labs.

Plausible but wrong. Restoration and label-free prediction produce images that look right. A prediction should be checked against measurements on real fluorescence before it supports a biological claim.

Batch effects. In Cell Painting, plates run on different days or at different sites can differ more than the treatments do. The JUMP consortium, for example, included a positive-control plate of 302 diverse compounds in every batch for this reason.

What this changes for biology

The practical effects are modest in description and large in scale. Segmentation that once required weeks of manual outlining can run overnight, so experiments that were too large to analyze become feasible. Live imaging with less light means longer observation of delicate cells such as stem cells and embryos. Public image datasets let small labs benchmark methods against industrial-scale data. These shifts connect to other uses of vision models in biology, including cancer-detection AI on pathology and radiology images, organoid models that produce complex 3D structures needing 3D segmentation, and deep learning for materials microscopy. On the molecular side, AlphaFold predicts structure from sequence, and imaging models extend that view up to the scale of whole cells. A parallel line of work decodes neural signals instead of cell photos, as described in neural networks decoding the brain and brain-computer interfaces. Electron microscopes, covered in how the electron microscope sees tiny things, are the other major image source now feeding these models.

The honest limit is that images show correlates of biology and not mechanism. A profile that matches a known drug is a lead, and a restored image is an estimate. The field's best results come when imaging is paired with orthogonal measurements, such as sequencing or biochemical assays, and when a model's performance is checked on data it has never seen.

Common confusions

"AI image enhancement reveals real detail." Restoration networks add plausible detail learned from training examples. A structure that appears only after enhancement is a hypothesis, to be confirmed by an independent measurement.

"A segmentation model works on any cell." It works on cells that resemble its training data. Generalist models narrow the gap, and the Cell Tracking Challenge analysis shows the gap persists.

"Cell Painting is a machine-learning method." It is a staining and imaging assay. Machine learning is used to analyze its images, but the profiles come from the dyes, the microscope, and the feature extraction.

"Bigger datasets mean better biology." More images help only if they sample the variation the model will face. Millions of images from one cell line and one microscope still teach the model one narrow world.

The takeaway

Neural networks have changed microscopy by solving, or nearly solving, the bottleneck steps: finding cells reliably, recovering usable images from less light, predicting labels that would otherwise need a stain, and condensing cell appearance into profiles that can be compared at scale. The tools are open source, the benchmarks are public, and the best-tested ones, such as Cellpose, U-Net variants, and CARE, rest on peer-reviewed validation. The caution is just as clear in the benchmark data: these models are specialists, and they perform best on conditions like the ones they were trained on. A good rule for reading any claim in this area is to ask what the model was trained on, what it was tested on, and whether anyone checked the result against the real specimen.

FAQ

Q: What is cell segmentation and why does it matter? A: Segmentation assigns each pixel in a microscopy image to a particular cell or structure, or to background. It is the first step in most quantitative analysis, because measurements such as cell size, brightness, or organelle count require knowing which pixels belong to which cell. Errors at this step carry through to every later result.

Q: What is Cell Painting and how is it used in drug discovery? A: Cell Painting is a microscopy assay that uses six fluorescent dyes, imaged in five channels, to label eight cellular components. Software measures about 1,500 features per cell to build a morphological profile. In drug discovery, profiles are compared across compounds to group those with similar effects and to flag unexpected activity, generating hypotheses for follow-up experiments.

Q: How does AI make microscopy images look better than the raw data? A: Restoration networks such as CARE learn from pairs of low-quality and high-quality images of the same specimen, then predict a cleaner image from a new noisy one. They do not recover information that was never recorded. They fill in likely structure based on training examples, so results should be validated before they are quantified.

Q: Can AI microscopy tools work on images that were not in their training data? A: Sometimes, but performance often drops. The 2023 Cell Tracking Challenge analysis found that pretrained methods decreased significantly in performance on different morphologies or imaging modalities. Generalist models trained on diverse data narrow the gap, and fine-tuning on a small set of labeled images from the new setting commonly helps.

Q: What is label-free prediction? A: It is a method that predicts fluorescence images from unlabeled transmitted-light images using a network trained on paired examples. A 2018 study from the Allen Institute for Cell Science showed this for many cell structures, avoiding the stains and light exposure that fluorescence needs. It performed poorly for some structures, such as desmosomes and actomyosin bundles.

Q: Which software do biologists use for AI-based image analysis? A: Commonly cited open-source tools include Cellpose and StarDist for segmentation, CARE for restoration, napari as a viewer for plugins such as micro-SAM, Fiji for general image analysis, and CellProfiler for feature extraction, which the Cell Painting protocol uses. The best choice depends on the cell type, imaging method, and whether you need 2D or 3D analysis.

Read next: