The McGurk Effect: When Eyes Override Ears
Photo: Google DeepMind / Pexels
You are watching a video. The audio track is playing the syllable "ba." The speaker's lips are visibly forming "ga." What you hear is "da" — a sound that exists in neither the audio nor the video. Your brain invented it. This is the McGurk effect, documented in a single two-page paper published in Nature in 1976, and it has not stopped forcing scientists to revise their assumptions about speech, perception, and how the senses work together ever since.
What McGurk and MacDonald Were Actually Studying
Harry McGurk was a developmental psychologist at the University of Surrey working on an unremarkable problem: how do infants learn to recognize speech? His research assistant John MacDonald was preparing dubbed video stimuli — recordings where the audio track of one phoneme was laid over video of a speaker producing a different phoneme — to study whether very young children could tell the difference.
The discovery that followed was, by McGurk's own later account, entirely accidental. When MacDonald dubbed an audio "ba" over a video of lips forming "ga," neither of them heard "ba." They heard "da." They ran it again. Still "da." They checked the equipment. The audio was definitely "ba." The video was definitely "ga." But what they perceived was something that was not present in either stream.
Curious enough to treat the accident as an experiment, they systematically tested the phenomenon across multiple phoneme combinations, multiple talkers, and multiple listeners. The results were consistent enough to constitute a new finding, not an equipment glitch, and they wrote it up and submitted it to Nature.
Source: McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746–748.
The Setup
The canonical McGurk stimulus is straightforward to describe and, if you have not encountered it before, astonishing to experience.
A video is recorded of a speaker producing the syllable "ga" — the velar stop, formed by pressing the back of the tongue against the soft palate. The audio track of that video is replaced with a recording of the same speaker saying "ba" — the bilabial stop, formed by pressing both lips together. The combined stimulus is played to a listener.
The three possible phonemes at play here differ in where in the vocal tract they are produced:
- "ba": bilabial — lips together (visible on the face)
- "ga": velar — back of the tongue against soft palate (invisible from outside)
- "da": alveolar — tongue tip touching the ridge behind the upper front teeth (moderately visible)
The brain, receiving conflicting signals, does not arbitrate by defaulting to one sense or averaging them arithmetically. It constructs a third percept — "da" — that is plausibly compatible with both the auditory and visual information simultaneously. The lip movement seen in the video is inconsistent with "ba" (since "ba" requires lip contact, which the "ga" video does not show). The audio is inconsistent with "ga." The brain's solution is to synthesize a phoneme that could explain both inputs.
This is not a failure of perception. It is perception working exactly as it was built to — integrating all available information into the most coherent single interpretation it can construct.
Source: McGurk Effect — Britannica
The Protocol
McGurk and MacDonald's original study was modest by modern standards. They presented dubbed stimuli to adults and to children of different ages and recorded what participants reported hearing. The stimuli covered several audiovisual combinations:
- Audio /ba/ + Visual /ga/ → Perceived /da/ or /tha/ (the dominant McGurk response)
- Audio /ga/ + Visual /ba/ → Perceived /bga/ or /gba/ (a different fusion response)
- Congruent stimuli (audio and visual matched) → Correct identification
The key controls were the reverse conditions — where audio and visual were presented without conflict — which confirmed that participants could accurately identify both the auditory and visual phonemes when shown in isolation. The illusion was specifically a product of the combination.
They also tested the effect with eyes closed (removing the visual input), which immediately resolved the percept back to "ba." Open eyes restored "da." The manipulation could be switched on and off by controlling visual access, which was precisely the kind of stimulus control needed to establish causality.
The Results
The core findings were clean. Adults overwhelmingly perceived the fused "da" response when presented with audio /ba/ + visual /ga/. Children showed the effect too, though its strength varied with age — younger children were somewhat less susceptible, suggesting that the audiovisual binding mechanism strengthens with experience in speech perception.
Three findings in particular made the paper significant beyond its immediate finding:
First, the illusion was resistant to knowledge. Once participants were told what was happening — that the audio and video had been dubbed separately and did not match — they still perceived "da." Understanding the trick intellectually did not break it. This placed the integration mechanism at a pre-conscious stage of sensory processing, not at a level where top-down knowledge could intervene.
Second, the integration was not simply "take the visual." The fused percept "da" is not what you would produce if you simply lip-read and ignored the audio. Lip-reading alone from "ga" would typically produce "ga," or possibly something ambiguous. The result was a genuine synthesis of both signals, producing a phoneme consistent with both.
Third, the effect generalized across talkers, languages, and recording conditions. This was not an artifact of one specific speaker's face or one particular recording setup. It was a general property of audiovisual speech perception.
Source: McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746–748.
The paper has been cited more than 4,800 times, remarkable for a two-page article, and it gave rise to an entire subfield of multisensory speech research that continues to generate hundreds of publications a year.
Source: Audiovisual speech perception: Moving beyond McGurk — PMC
The Timeline
| Year | Event |
|---|---|
| 1976 | McGurk & MacDonald publish "Hearing Lips and Seeing Voices" in Nature |
| 1980s | Effect confirmed in multiple independent laboratories; extended to other phoneme pairs |
| 1990s | Neuroimaging begins to identify brain regions involved in audiovisual speech integration (STS) |
| 2000s | Cross-cultural studies reveal variation in McGurk susceptibility across languages |
| 2005 | Studies document reduced McGurk susceptibility in native Japanese speakers |
| 2010s | Effect used in cochlear implant rehabilitation research and autism spectrum disorder studies |
| 2021 | Rethinking paper in Attention, Perception, & Psychophysics examines the illusion framing |
The Controversy
The McGurk effect looks like a simple perceptual fact. It is not. Several controversies have accumulated around it.
Cross-cultural variability. Early work assumed the effect was universal. Later studies found that native Japanese speakers show significantly lower McGurk susceptibility than English speakers. Several explanations have been proposed. One is phonological: Japanese phonology places less communicative weight on place of articulation distinctions that English relies on, so the brain of a Japanese speaker may weight visual lip information differently. Another is attentional: English listeners are trained from infancy in a visually richer speech environment because English has more minimal pairs distinguished by place of articulation, making visual information more informative. A 2023 study investigating Finnish and Japanese speakers found that the effect is strongly language-dependent, with specific phoneme combinations showing different fusion rates across linguistic backgrounds.
Is it really an illusion? A 2021 paper in Attention, Perception, & Psychophysics challenged the "illusion" framing directly. The authors argued that calling the fused percept an illusion implies it is a mistake — that one input is "right" and the fusion response is "wrong." But if the brain's job is to infer what speech is present in the environment, and both auditory and visual cues are legitimate sources of information, then a fused response that integrates both coherently is not an error. It is the correct behavior for a system optimizing over noisy, multi-channel sensory data. The "illusion" label, they argued, imports a normative judgment that does not belong.
Source: Rethinking the McGurk Effect as a Perceptual Illusion — Attention, Perception, & Psychophysics, 2021
Individual differences are large. Unlike many classic perceptual effects, the McGurk effect shows substantial variability across individuals, even within the same language group. Some people consistently hear "da" on the standard stimulus; others reliably hear "ba." These individual differences appear to be partly heritable and partly related to how much a person naturally uses visual speech cues in everyday communication. The reasons for this variability are not fully understood, and it complicates any simple claim that the effect tells us something universal about human perception.
Source: McGurk Effect — Wikipedia
Replication and Reanalysis
Unlike some classic psychology experiments, the McGurk effect has replicated consistently across the decades and across cultures — though the effect size varies. The basic phenomenon — that audio /ba/ + visual /ga/ produces a fused percept distinct from either input — is one of the most robust findings in perceptual psychology.
What has required more careful treatment is the claim that the effect is entirely pre-conscious and impervious to top-down influence. Several studies have shown that while explicit knowledge does not eliminate the illusion (you cannot just decide to hear "ba" by knowing the audio is "ba"), attentional manipulations can modulate it. If you direct a listener's attention specifically to their auditory stream and away from the video, the visual influence on perception weakens somewhat. The effect operates below the level of voluntary choice but is not entirely sealed off from attentional modulation.
Source: Audiovisual speech perception: Moving beyond McGurk — PMC
Neuroimaging work has located the integration mechanism primarily in the superior temporal sulcus (STS) — a region that receives input from both auditory and visual cortex and is active during audiovisual speech processing. Lesion studies and TMS work have supported the STS's role as a critical convergence zone where the fusion happens.
Where It Stands Today
Cochlear implant rehabilitation. Cochlear implants restore a form of hearing to profoundly deaf individuals, but the signal quality is poor compared to normal acoustic hearing — the implant converts sound to a small number of electrical stimulation channels rather than the thousands of frequency-specific hair cells in a healthy cochlea. In this context, visual speech cues become more important, and the McGurk effect is used to probe how well cochlear implant users are integrating audiovisual speech information. Research has found that cochlear implant users often show stronger McGurk susceptibility than normal-hearing listeners, suggesting the brain upweights visual input when auditory input is degraded.
Autism spectrum disorder research. A consistent finding across multiple studies is that autistic individuals show reduced McGurk susceptibility compared to neurotypical controls. This aligns with broader evidence of atypical multisensory integration in autism — the autistic brain appears to rely less on visual speech cues and weight sensory streams more independently rather than fusing them. Whether this reflects a deficit in audiovisual integration circuitry or an adaptive strategy that avoids over-reliance on ambiguous visual signals is a matter of ongoing debate.
AI and multimodal models. The McGurk effect has become a benchmark for evaluating audiovisual speech recognition systems and multimodal large language models. A system that integrates audio and video for speech recognition faces the same computational problem that the human brain does: how do you weight conflicting signals from two sensory channels? Human McGurk susceptibility patterns provide a reference distribution against which machine behavior can be compared, and the effect is used as a probe for whether artificial systems are performing human-like sensory fusion or simply ignoring one modality.
The fundamental insight. The McGurk effect established that speech perception is not hearing plus optional visual monitoring. Visual information is built into the speech perception process at a level where it cannot easily be detached. Your lips, as a listener, are part of what you hear with. That fact has shaped everything from hearing aid design to how video conferencing systems are engineered, and it still generates new questions every year.
Source: Hearing Lips and Seeing Voices: Origins and Development — Multisensory Research, 2018
Neuroscience of the Illusion: Where Audio and Visual Speech Merge
Superior temporal sulcus (STS) — a fold on the outer surface of the temporal lobe where auditory and visual cortex projections converge, making it the primary site for audiovisual speech integration.
For decades, researchers could observe the McGurk effect behaviorally but not watch it happen inside the brain. Functional MRI changed that. When a listener hears a McGurk stimulus and reports the fused percept "da," the posterior STS activates significantly more than it does when the audio and video match. Crucially, when the same listener closes their eyes and hears only the "ba" audio — breaking the illusion — posterior STS activation drops. The region tracks the fusion event, not just the presence of speech.
Source: Calvert GA, Bullmore ET, Brammer MJ, et al. "Activation of auditory cortex during silent lipreading." Science 276(5314): 593–596 (1997). https://www.science.org/doi/10.1126/science.276.5314.593
The STS is not working in isolation. It receives processed auditory input from the primary auditory cortex (A1) and visual motion and form information from the superior temporal polysensory area (STP) and visual association cortex. Think of those upstream areas as separate instrument tracks in a recording studio — each processes its own channel — while the posterior STS is the mixing desk where the two tracks are combined into a single output. When the tracks conflict, the STS does not mute one and amplify the other; it blends them into a new signal consistent with both.
Transcranial magnetic stimulation (TMS) work has helped confirm causality. Briefly disrupting the STS with TMS reduces — though does not entirely eliminate — McGurk susceptibility in the same subjects, while disrupting primary auditory cortex does not change susceptibility in the same way. This dissociation supports the STS as the locus of integration rather than earlier stages of auditory processing.
Source: Beauchamp MS, Nath AR, Pasalar S. "fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect." Journal of Neuroscience 30(7): 2414–2417 (2010). https://www.jneurosci.org/content/30/7/2414
The effect varies considerably across individuals, and structural MRI studies have begun to explain why. Listeners with greater cortical thickness in the posterior STS region show stronger McGurk susceptibility, suggesting that individual differences in the integration mechanism partly reflect differences in the underlying neural architecture — not just attention or experience. This is not a small effect: in some studies, the range of McGurk susceptibility across participants spans the full spectrum from near-zero to near-100%, and the structural correlates account for a meaningful portion of that variance.
Cross-language variation adds another layer. Native Japanese speakers consistently show lower McGurk susceptibility than English speakers on standard stimuli, and neuroimaging comparisons suggest that the Japanese-speaking brain recruits the STS differently during audiovisual speech processing. English phonology relies heavily on the visual contrast between bilabial stops ("b," "p") and other stop consonants, giving the visual channel greater discriminative value. Japanese phonological structure makes those same visual contrasts less informative, so the STS in Japanese-experienced brains appears to assign less weight to facial movement cues when computing the auditory percept.
Source: Sekiyama K, Kanno I, Miura S, Sugita Y. "Auditory-visual speech perception examined by fMRI and PET." Neuroscience Research 47(3): 277–287 (2003). https://www.sciencedirect.com/science/article/abs/pii/S0168010203002265
What this neuroscience reveals is that the McGurk effect is not a quirk of the peripheral senses — it is a product of learned cortical weighting. The brain does not apply a fixed formula to audiovisual speech; it uses a prior shaped by years of experience with a specific language's phonological structure. The STS is not a universal speech-fusion machine. It is a machine calibrated, over a lifetime, by the sounds and lip movements most predictive in the listener's native tongue.
FAQs
Q: Why do I hear "da" instead of either "ba" or "ga"? A: Your brain receives conflicting information: the auditory signal is compatible with "ba" (a lip-produced sound), but the visual lip movement is incompatible with "ba" (since "ba" requires the lips to touch, and the "ga" video does not show that). Rather than accept either input as the final answer, the brain synthesizes a phoneme — "da" — that is plausibly consistent with both streams simultaneously. It is a form of optimal inference under conflicting evidence.
Q: Can you make yourself stop perceiving "da" once you know about the effect? A: No. Knowing that the audio and video are mismatched does not break the illusion for most people. You can sometimes hear "ba" if you close your eyes, but as soon as you open them and see the lips forming "ga," the fusion response returns. This resistance to top-down correction is one of the effect's most important properties.
Q: Do people who are deaf or hard of hearing experience the McGurk effect? A: It depends on their hearing history and whether they use cochlear implants. Profoundly deaf individuals who rely on lip-reading often show different patterns of audiovisual integration. Cochlear implant users, who receive a degraded auditory signal, often show stronger McGurk susceptibility than normal-hearing listeners because their brain upweights visual information to compensate for the poor-quality audio.
Q: Why do Japanese speakers show less susceptibility to the McGurk effect? A: Several hypotheses exist. One focuses on the phonological structure of Japanese, which has fewer minimal pairs distinguished by place of articulation, making visual lip information less informative for disambiguation. Another focuses on different patterns of attention to the face during speech in Japanese versus English-speaking communities. Current research suggests both phonological and cultural-attentional factors contribute to the difference.
Q: Is the McGurk effect a flaw in human perception? A: That depends on how you define a flaw. The illusion arises because the brain is doing something principled: integrating all available sensory information into the most coherent interpretation. In natural settings, audio and visual speech almost always match, and using both together improves accuracy significantly over relying on either alone. The illusion only appears when experimenters artificially mismatch the streams, a situation the perceptual system was not built to detect.
Q: Where in the brain does audiovisual speech fusion happen? A: Neuroimaging studies consistently implicate the superior temporal sulcus (STS), particularly its posterior portion, as a key convergence zone for audiovisual speech information. The STS receives projections from both auditory and visual processing areas and its activation increases when participants perceive a McGurk fusion response compared to when they perceive the auditory phoneme alone.
The Takeaway
A two-page accident of a paper from 1976 proved that the human mind does not passively receive sounds and images — it actively synthesizes them into unified percepts, sometimes constructing things that are in neither the audio nor the video. Speech perception is inherently multisensory, and the visual information your brain extracts from a speaker's face is not supplementary to what your ears do; it is woven into the process at a level where conscious knowledge cannot easily separate them. That insight now shapes cochlear implant rehabilitation, autism research, and the design of AI systems that must parse human communication — from a study that began as a mistake with a dubbing machine.
Read next:
Weekly digest
Stay ahead of the curve
Get the best science, AI, and tech stories delivered to your inbox every week. No spam — unsubscribe any time.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. Articles are drafted against peer-reviewed literature and primary sources, fact-checked, and edited by a human before publication. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles
Mendel's Pea Plant Experiments: How Genetics Was Born
Between 1856 and 1863, Gregor Mendel cross-bred 30,000 pea plants and found the mathematical rules of heredity. Here is what he did and why it still defines genetics.
The Marshmallow Experiment: Self-Control, Class, and a Crumbling Prediction
Walter Mischel's 1972 marshmallow test predicted life outcomes from a single moment of self-control. A 2018 replication of 918 children changed the story.
How Pain Works in the Brain: It's Not What You Think
Pain is not a signal your body sends to your brain. It is a decision your brain makes — and that distinction has transformed medicine.
Pavlov's Dogs: How a Digestion Study Rewired Psychology
Pavlov's dogs experiment began as digestion research, not psychology. Learn the real setup, the bell myth, the Tower of Silence, and how it shapes PTSD therapy today.
The Iowa Gambling Task: How Emotions Drive Decisions
How a 1994 card game revealed that gut feelings guide rational decisions. The Iowa Gambling Task and the somatic marker hypothesis, explained.
The Libet Experiment: Did Your Brain Decide Before You Did?
In 1983, Benjamin Libet found brain activity began 350ms before conscious intention to move. The free will debate that followed has been partially rewritten by modern neuroscience.