Neural Networks Decoding the Brain: How AI Reads Neural Activity

In May 2023, a University of Texas at Austin team reported in Nature Neuroscience that a language-model-based decoder could turn fMRI brain scans into running text capturing the gist of stories people were hearing, with no surgery. Each of the three participants had spent 16 hours in a scanner so the system could learn their individual brain. This article explains how brain decoding works, what recent studies achieved, and where headlines run ahead of the data.

Key takeaways

A brain decoder is a statistical model trained on paired data: recordings of brain activity plus the words, images, or movements that accompanied them.
Implanted electrodes give far better signals than scanners, and trial participants with paralysis have reached speeds of 62 to 78 words per minute with decoders.
Non-invasive decoders recover meaning and approximate images, not verbatim thoughts, and they need many hours of training data from each person.
Cooperation matters: the Texas study found that the person must take part both in training the decoder and in applying it.
Privacy debates are already shaping policy, including a 2023 ruling by Chile's Supreme Court on an EEG headset maker.

The basic idea: learn the map, then run it backward

The brain represents the world as patterns of activity spread across millions of neurons. When you hear a word, see a face, or try to move your hand, reproducible patterns show up in particular regions. Language areas respond to speech, visual cortex to images, motor cortex to intended movement. A decoder's job is to learn the mapping between those patterns and their content, then run it in reverse: given a new pattern, say what most likely caused it.

Think of it as translating between two languages when you only have a stack of bilingual examples and no dictionary. With enough paired sentences, a model can infer which brain patterns tend to go with which meanings, even though no one can write the rules by hand. That is the role of a neural network here. It is a flexible pattern-matching system, introduced in how neural networks work, tuned on examples until its predictions line up with reality.

Encoding models and decoding models

Researchers distinguish two directions. An encoding model predicts the brain's response to a stimulus: given a sentence, what would the activity look like? A decoding model goes the other way: given the activity, what was the stimulus? Many of the best systems combine both. The decoder proposes candidates, and an encoding model checks how well each candidate would explain the recording. It resembles a detective who guesses a suspect, then tests whether that suspect would have left the evidence found at the scene.

The foundation for this approach was laid by work that mapped meaning across the cortex. In a 2016 Nature study, Alexander Huth, Jack Gallant, and colleagues scanned seven people as they listened to more than two hours of stories from The Moth Radio Hour and built voxel-by-voxel models of what each patch of cortex responded to. The resulting semantic atlas showed that areas of cortex represent groups of related concepts in patterns that look broadly consistent across individuals, although the authors noted substantial individual differences (Gallant Lab summary). Those individual differences are one reason decoders need training data from each person.

What the instruments can and cannot see

The choice of recording method determines what can be decoded. No current method is both sharp in space and fast in time without surgery, so every approach trades one for the other.

Method	What it measures	Strength	Main limit
fMRI	Blood-oxygen changes that follow neural activity	Fine spatial detail across the whole brain	Slow; samples about every 2 seconds and blurs fast changes
EEG	Electrical activity at the scalp	Millisecond timing, cheap, portable	Blurry spatial detail; skull smears the signal
MEG	Magnetic fields from neural currents	Millisecond timing with better spatial detail than EEG	Needs a shielded room and a very large machine
Implanted electrodes	Voltage near neurons, on or in the cortex	Highest signal quality	Requires brain surgery

fMRI (functional magnetic resonance imaging) does not measure neurons firing. It tracks changes in blood oxygen that follow neural activity, which makes it a slow, smeared proxy. A 2023 Meta AI paper describes the trade in numbers: fMRI is sampled at about 0.5 Hz, meaning one whole-brain image every two seconds, while MEG resolves activity at about 5,000 Hz (Benchetrit, Banville, and King). For deeper background on the scanner, see fMRI versus MRI. EEG, the electrode cap that dates back to a psychiatrist's early experiments, is covered in the history of the EEG.

Why the blur matters for language

Natural conversation runs at about 160 words per minute, or roughly 2.7 words per second, while fMRI delivers one picture every two seconds. A single scan therefore reflects many words blended together. The Texas team solved this with a language model. Instead of reading words directly from the scan, the system generated candidate word sequences and used an encoding model to predict how each would look in the brain, keeping the candidates whose predicted activity best matched the recording. The slow measurement still limits what can be recovered, but the language model supplies the fine structure the scanner cannot.

From simple categories to reconstructed movies

The step change in visual decoding came from modeling how the brain responds to natural stimuli. In 2011, Shinji Nishimoto, Jack Gallant, and colleagues published in Current Biology a method that reconstructed approximate movie clips from visual cortex activity. Their motion-energy encoding model separated fast visual information from the slow blood-flow signal, and a Bayesian decoder searched a large library of natural movie clips for the best matches. The reconstructions were blurry and approximate, but they showed that dynamic natural vision could be decoded with ordinary fMRI (Gallant Lab).

The leap since then comes from pairing brain data with large pretrained models. Visual cortex is one of the best-understood regions of the brain, a story that begins with Hubel and Wiesel's cat experiments, and image models supply a rich description of what an image contains.

Stable Diffusion meets fMRI

In 2023, Yu Takagi and Shinji Nishimoto, then at Osaka University, presented work at the CVPR conference that used a latent diffusion model, the Stable Diffusion image generator, to reconstruct high-resolution images from fMRI. They reported that the approach worked without training or fine-tuning complex deep-learning models, because only simple linear mappings from brain activity into the generator's internal representations were fitted (CVPR 2023 paper page). They also examined which parts of the diffusion model corresponded to different brain functions. For a primer on the generator, see how diffusion models work.

The approach produces images that look photographic because the generator is good at producing realistic pictures. That cuts both ways. A reconstruction can look convincing while getting details wrong, since the generator fills in plausible content. A reader should treat the images as informed guesses, not as photographs taken from the mind.

Faster signals with MEG

Meta AI researchers pursued a different route: MEG instead of fMRI, to track perception as it unfolds. Their decoder combined pretrained image embeddings, a module trained end to end on MEG, and an image generator. The authors reported a sevenfold improvement in image retrieval over classic linear decoders, found that late brain responses were best decoded using the DINOv2 vision model, and concluded that MEG mainly captures high-level visual features, whereas 7-tesla fMRI also recovers low-level ones. They called the results preliminary (arXiv 2310.19812). The goal is a system that follows what someone sees in near real time, which would need the temporal precision fMRI lacks.

Decoding speech from scalp recordings

Another Meta group, led by Alexandre Défossez and Jean-Rémi King, applied self-supervised speech models to non-invasive recordings. Published in Nature Machine Intelligence in 2023, the work used data from 175 volunteers across four public datasets. From just 3 seconds of MEG signal, the model could pick out the matching speech segment from more than 1,000 candidates with up to 41 percent accuracy on average across participants, and above 80 percent for the best participants. Accuracy on EEG was much lower. The model relied on wav2vec 2.0, a speech representation system, to define a target for the brain signals to predict.

Notice what the task is. The system does not generate the sentence from scratch. It chooses among known options, a "which of these 1,000 clips was this person hearing" test. That makes it a useful research benchmark, though not a mind-reading device. The numbers also depend on averages: some people produce signals that decode well, and others do not.

How a language model turned fMRI into text

The Texas study, led by Jerry Tang and Alexander Huth, is the clearest example of why this field is both impressive and constrained. A language model (the paper used an early GPT model) proposed word sequences. An encoding model, trained on each participant's own scans, predicted brain responses for each candidate. The system kept sequences whose predicted responses matched the measured ones and discarded the rest, building up text one segment at a time.

The output captured meaning rather than exact wording. Researchers who reviewed the work for the Science Media Centre noted that the decoded text was not always accurate and often contained errors, and that it did not transfer across people, since a decoder trained on one subject performs poorly on another. They also pointed out that the work used just three participants, who each listened to 16 hours of stories in the scanner, and that results for imagined stories and silent movies were exploratory and less reliable (Science Media Centre).

The cooperation requirement

The authors tested whether the decoder could be used without the person's cooperation and reported that cooperation was required both to train the decoder and to apply it. That implies a person who deliberately attends to something else can withhold the signal the decoder needs. This is a real safeguard for fMRI, because a scanner needs a still, willing participant for hours. It is not a permanent guarantee. Implanted systems and future non-invasive devices may behave differently, and researchers have argued that the larger privacy risks come from invasive implants and from consumer devices rather than from scanners.

Implanted decoders: where the speed is

Electrodes placed on or in the cortex record from small groups of neurons and give much cleaner signals. The studies below involve people who cannot speak because of stroke, spinal cord injury, or amyotrophic lateral sclerosis (ALS, a disease that destroys motor neurons). Each is a clinical-trial result from one or a few participants, not an approved product.

Study	Participant and method	Result
Moses et al., NEJM, 2021	Person with brainstem stroke; electrode array on the speech cortex; 50-word set	Median 15.2 words per minute; median word error rate 25.6%
Willett et al., Nature, 2021	Person with spinal cord injury; imagined handwriting decoded from motor cortex	90 characters per minute at 94.1% raw accuracy; over 99% offline with autocorrect
Willett et al., Nature, 2023	Intracortical arrays; attempted speech	62 words per minute, about 3.4 times the prior record
Metzger et al., Nature, 2023	47-year-old woman with brainstem stroke; 253-channel surface array	Median 78 words per minute; median word error rate about 25%; speech synthesis and a facial avatar
Card et al., NEJM, 2024	45-year-old man with ALS; four arrays with 256 electrodes	97.5% accuracy sustained over 8.4 months; about 32 words per minute; 125,000-word vocabulary

For comparison, the Stanford team puts natural conversation at about 160 words per minute, so even the fastest result is under half of normal speed. The same 2023 paper reported word error rates of 9.1 percent on a 50-word vocabulary and 23.8 percent on a 125,000-word vocabulary, the first large-vocabulary demonstration the authors knew of (Willett et al., 2023).

What the table shows about progress

Speed and vocabulary have climbed together. The 2021 NEJM system recognized only 50 words and was about three times faster than the typing interface the participant normally used. The 2023 systems moved to vocabularies of hundreds or thousands of words, with the Metzger system using 1,024 words for real-time decoding and offline tests up to 39,378 words. The 2024 Card study reached a 125,000-word vocabulary, required only 30 minutes of calibration recordings to start, and ran for more than 248 cumulative hours of use (PubMed record as summarized in later reports).

Several caveats hold across these studies. Each involved one participant. Accuracy figures depend heavily on vocabulary size and on language-model correction, which can fix an error but can also steer a decoded sentence toward the likely wording instead of what the person intended. And these systems decode attempted speech, meaning the person tries to move their mouth and voice, not silent thoughts.

Handwriting and the surprise of motor cortex

The handwriting study deserves a note. The participant imagined writing letters by hand on ruled paper, and a recurrent neural network read the resulting activity in motor cortex. The system outpaced earlier point-and-click interfaces, which peaked at about 40 characters per minute, and approached the roughly 115 characters per minute the authors estimated for smartphone typing in the participant's age group (Willett et al.). The finding suggested that neural patterns for distinct handwritten letters remain distinguishable years after paralysis.

Devices in trials

Several companies are developing implants intended for everyday use. The most reliable evidence is the peer-reviewed literature, so treat company claims with care.

Synchron uses a stent-mounted electrode array, the Stentrode, delivered through a blood vessel to sit near the motor cortex without open-brain surgery. Its SWITCH study in Australia involved four people with severe paralysis. Reports on the study, which appeared in JAMA Neurology, said all four completed 12 months of follow-up without persistent neurological deficits or device migration, and each used the system to control a computing device. Most coverage came from the company, and four participants is a very small sample.

Neuralink implanted its first human participant, Noland Arbaugh, in January 2024 as part of its PRIME study. He later used the implant to control a laptop cursor. Within weeks, a number of the electrode-bearing threads retracted from his brain, and the company said it made the implant more sensitive to compensate. The case shows the engineering problem every implant faces: the brain is soft, it moves, and the body reacts to foreign objects. For deeper coverage see how Neuralink works and brain-computer interfaces and AI.

Inner speech and the privacy question

If a decoder can read attempted speech, can it read speech you did not intend to share? A 2025 Cell study from Stanford tested this with four participants who already had implanted arrays. Imagined speech produces motor-cortex signals similar to attempted speech but fainter, and with help from machine learning the team decoded sentences from a 125,000-word vocabulary with accuracy of up to 74 percent. Because inner speech could leak out during normal use, the researchers built in a mental password: the decoder stayed silent until the participant imagined the phrase "Chitty Chitty Bang Bang." They also trained the system to ignore inner speech altogether, which worked but cost speed and ease (NPR report).

The study shows engineers designing privacy into the device rather than treating it as an afterthought. It also shows why the question is no longer abstract.

Neurorights in law

Policy has started to move. Chile amended its constitution through Law 21.383 in 2021 to say that the law will regulate the use of scientific and technological development with particular emphasis on safeguarding brain activity and the information derived from it. In August 2023, Chile's Supreme Court ruled in a case brought by Guido Girardi Lavin against Emotiv Inc., the maker of the Insight EEG headset. The court treated neural data as sensitive and biometric data and stressed informed, specific consent for research use (Chilean Supreme Court ruling analysis). Commentators disagree about how far the ruling reaches, and some argue that existing data-protection law is a poor fit for neural data.

Neuroscientist Rafael Yuste and the Neurorights Foundation, which he founded at Columbia University, have pushed for international frameworks to protect mental privacy. Scholars still debate whether new rights are needed or whether existing privacy and medical-ethics law can be stretched to cover neural data. That disagreement is healthy at this stage, because the technology is changing faster than the legal categories.

What decoding cannot do

Popular coverage of this field often blurs several distinct claims. Here is what the evidence supports and what it does not.

It works on trained individuals, in the lab. Each decoder is calibrated to one person's brain. Researchers reviewing the Texas work stressed that decoders do not transfer across people.
It decodes meaning or attempts, not hidden thoughts. Most success comes from attempted speech, which is intentional, or from perceived stimuli, which are externally defined. Private, unexpressed thoughts are the hardest case.
Language models do part of the work. A decoder can seem to recover a sentence because the language model supplies likely words. This makes evaluation tricky, and good studies compare against baselines that use the language model alone.
Fast measurement and good spatial detail do not coexist non-invasively. fMRI's blur and EEG's smear are physical constraints, not just engineering problems.
Small samples limit confidence. Many headline results involve one to seven participants, so effect sizes in the wider population remain unknown.

A useful comparison is AI medical-scan analysis. In both settings, models find statistical regularities in noisy biological signals, and in both, performance on a carefully curated lab dataset says little about performance in messy conditions until it is tested there.

Common confusions

"AI can read my mind." Not in the popular sense. The Texas decoder could not work without hours of the person's cooperation, and decoders for implants require surgery. Thought experiments about covert reading are explored in what if AI could read your mind, but present capability is far from that scenario.

"Higher accuracy means it understands you." Accuracy is measured against a defined task: choose the right clip, recover the right word from a vocabulary. A decoder can score well on that task without anything resembling comprehension. It is statistics, not understanding.

"The reconstructed image is what the person saw." It is a plausible image consistent with the decoded features. Image generators fill in detail that the brain data cannot supply, so reconstructions should be judged by whether they match the real stimulus, not by whether they look sharp.

"Decoding is new." The principle has deep roots. Studies of how neurons carry signals, including the Hodgkin-Huxley work on the action potential, and decades of work on brain mapping and neural circuits laid the groundwork. What is new is the scale of data and the power of pretrained models, which lean on ideas such as transformer attention and embeddings.

Why this matters for medicine

The strongest case for the technology is restoring communication. A person with ALS who loses speech can currently rely on eye-tracking keyboards, which are slow and tiring. The 2024 study in a participant with ALS reached about 32 words per minute with sustained high accuracy, and the participant used it in self-paced conversation. A faster, accurate decoder would not change the underlying disease, but it could change daily life.

Related work crosses into other parts of biology and machine learning. Imaging in cell biology shares many techniques with brain decoding, as in neural networks for cell imaging and microscopy, and the broader pattern of models finding structure in biological data appears in AI-driven antibiotic discovery. Researchers also study how neuroplasticity might help users learn to control devices, though that line of work is still early.

The takeaway

Brain decoding works by learning, from paired examples, how one person's brain responds to specific words, images, or intended movements, then running that map backward. The best results come from implanted electrodes in people who volunteered for trials, and they restore a useful but still partial version of speech: 62 to 78 words per minute in the fastest reports, against about 160 for natural conversation. Non-invasive decoders recover meaning and rough imagery after many hours of individual training, and they need the person's participation. The most useful way to read each new headline is to ask four questions: how many people, how many hours of training, what task was scored, and whether the person had to cooperate. Those answers separate a real advance for people who cannot speak from a demonstration that sounds like mind reading but is not.

FAQ

Q: Can AI currently read minds? A: No, not in the popular sense. Current systems decode attempted speech, perceived stimuli, or broad semantic content under controlled conditions, after hours of calibration on one person, and they make errors. The Texas fMRI study found that the person's cooperation was needed to train and to apply the decoder.

Q: How much data does a brain decoder need? A: It depends on the method. The 2023 fMRI language decoder used 16 hours of scanner time per participant. The Metzger speech decoder reached high performance with less than two weeks of data collection, and the Card ALS system needed 30 minutes of recordings to calibrate on its first day, then improved with further training.

Q: Which systems have been tested on people with paralysis? A: Implanted systems from several academic groups have been tested in clinical trials, including speech decoders at UC San Francisco and Stanford-linked groups, and device companies such as Synchron and Neuralink have begun human studies. The results described here come from small trials with one to a few participants each, and none is a general-purpose consumer product.

Q: What brain regions are easiest to decode? A: Sensory and motor regions are best understood because their activity maps closely onto external events: what you see, hear, or try to move. Speech motor cortex has proved especially useful for attempted-speech decoders. Abstract reasoning, which involves distributed networks, is much harder to decode reliably.

Q: Does brain decoding require surgery? A: Not always. fMRI, MEG, and EEG decoding are non-invasive, but they provide blurrier or slower signals and have so far recovered gist or chosen among candidates rather than producing fluent, accurate text. The fastest speech results come from surgically implanted electrodes.

Q: Can a decoder be used on someone without their knowledge? A: Current non-invasive methods make that impractical. The fMRI decoder needed many hours of the person's own training data and active cooperation, and a scanner is hard to conceal. Implants and consumer neural devices raise different concerns, which is why laws such as Chile's neurorights provisions and design features like the Stanford mental password exist.

Read next: