Kavli Institute for Brain and Mind
Search    
Find    

Research Profile

July 2009

Principal Investigator:Gene R. Stoner
Co-Investigators: Hulusi Kafaligonul, Ladan Shams

The Mechanisms Underlying Multisensory Interactions in the Brain

As we are all aware, objects and events encountered in our daily lives stimulate several of our senses at the same time. Generally, however, the processing within the brain of objects and events is segmented, with different brain areas responding to different sensory stimuli.

The temporal dynamics of different sensory modalities also differ. For example, the processing of sound within the ear is much faster than the processing of light within the eye. This means that the brain’s responses to a single auditory-visual event occur at different locations and in different timeframes. And learning just how these spatially- and temporally-distributed activities interact to create our unified multisensory experiences is one of the most fundamental problems in neuroscience.

Figure

Adapted from (Beauchamp, 2005)
Several recent studies have reported multisensory interactions within lower-level cortical areas thought to be sensory-specific. For instance, functional Magnetic Resonance Imaging (fMRI) scans suggest that activation in human cortical area MT+ -- widely accepted to underlie visual motion perception – is indeed modulated by auditory stimuli. But it is still not known what such activations reflect at the level of the single neuron.

Thus, the longterm goal of this team’s research is to understand the neuronal mechanisms underlying such cross-modal modulation. Their unusual experimental paradigm was inspired by an illusion developed just last year by U.K. scientists Elliot Freeman and Jon Driver. In the illusion, brief sounds systematically change the perceived direction of visual motion stimuli, although the sounds themselves provide no motion cues.

The illusion is believed to rely on the ability of sound to “capture” the timing of visual stimuli, a phenomenon known as temporal ventriloquism, and with use of their KIBM award the research team has successfully adapted it for use in certain neurophysiological experiments.

It seems that Freeman and Driver used “long-range” visual stimuli, involving large spatial and temporal displacements, and the neuronal basis of these is believed to involve higher-order processing than is associated with the “short-range” motion stimuli that activate area MT. Although it was believed unlikely that the F&D illusion would extend to short-range stimuli, the team happily discovered that it does. This has led to the hypothesis that area MT underlies this cross-modal illusion.

The team is now testing several other hypotheses about the way that auditory stimuli shift the timing of visual stimuli. And soon to start are neuronal recording experiments from macaque monkeys. This will be the first examination of auditory-visual perceptual interactions at the cellular level in area MT. By examining this neuronal data together with the human behavioral experiments, they envision being able to identify the mechanisms that underlie the amazing unified multisensory experience of the world that we all enjoy.

Past Research Profiles