Why Your Eyes Can Change What You Hear
Speech reaches the brain through both the ears and the eyes. A voice provides sound, while a speaker’s face offers clues about how that sound was formed. For example, some consonants look quite different on the lips even when they are hard to distinguish by sound alone. In ordinary conversation, those visual clues help the brain interpret speech, especially when the audio is unclear.
The Brain Combines Conflicting Clues
The McGurk effect occurs when a video shows one mouth movement while the soundtrack plays a different consonant. Instead of simply hearing the audio or noticing the mismatch, a listener may perceive a third sound—a blended interpretation that neither the mouth nor the soundtrack actually presents.
That happens because the brain generally treats speech as one event with both a visual and an auditory source. It weighs the clues together and settles on an interpretation, without waiting for conscious deliberation. When the sources conflict, that integration can produce a percept that feels like a sound coming from the recording itself.
A Small Change Can Alter the Perception
The effect depends on seeing the speaker’s mouth: cover it, and the listener may hear the sound differently. This makes the illusion useful for understanding how perception works—not as a simple recording of the outside world, but as the brain’s best interpretation of information arriving through several senses.
In conversation, that combination is usually helpful. The same process can also lead the ears astray: a visible mouth movement can change what a person hears, even when the soundtrack has not changed.
