why consider multisensory processing
- many situations involved bombardment of the sense
- combing information across senses can more quickly and accurately inform us about the outside world
what is multisensory integration
at a neuronal level it is a statistically significant difference between the number of impulses evoked by a crossmodal combination of stimuli compared to the number evoked by the most effective stimuli individually
multisensory enhancement
increased firing
multisensory depression
decreased firing
superadditive
overall (combination) multisensory is greater than sum of sensory inputs (e.g. auditory + vision)
- from distant
subadditive
- stronger unisensory when nearer
- multisensory combination is no greater or less than the two sensory inputs when summed together
weaker stimuli
more benefit of multimodal information when input from two weaker stimuli at the same time
Superior colliculus
- midbrain structure
- input visual pathway
- involved in orientating behaviour including saccadic eye movement to visual and auditory and somatosensory stimuli
MSi in the cat superior colliculus
- recordings from single neurones in cat SC
- cell responds to both visual and auditory alone (unisensory)
- responds more strongly to combination = superadditive
top-down cortical information
SC receives:
- sensory information from eye/ear
- information from cortical area (feedback)
- ‘unisensory’ processing areas can receive multisensory information
principles of multisensory integration
- inverse effectiveness
- temporal rule
- spatial rule
inverse effectiveness
strongest effect when individual cues are weak
temporal rule
strongest when occur at same (or similar) point in time
spatial rule
strongest when receptive fields overlap
(when stimuli come from same place)
multisensory receptive field
- cat SC
- independently manipulate location of auditory and visual stimuli to show they needed to be co-located to elicit a stronger response in a neuron
- overlapping receptive fields from two modalities
- animal can orient rapidly to location of stimulus
Visuo-tactile receptive fields
- studied in monkeys (cortical area not SC)
- visual and tactile RFs spatially aligned
visual RFs restricted to area surrounding the body - reduced response as visual stimulus moves further from the body
- Visual RFs are anchored to body part
- peripersonal space
peripersonal space
visual area around the body (where they can interact with other entities/ watch out for things)
remapping RFs
- cells in primate ventral pre-motor cortex
- regardless of where the animal was fixating, cells responded most to stimulus coming towards the tactile area of interest
- shows visuotactile receptive fields update / change with changes in body position and fixation
multisensory integration in humans
- fMRI imaging used to explore areas in the human brain
- e.g. trisensory in premotor cortex (audio,visual,tactile)
- different ‘rules’ may apply at different levels of processing e.g. in cortical areas semantic (meaning) for processing speech and movement may become more important
speech perception
visual information can
- improve speech perception e.g. seeing lip movements in a noisy environment
- influence where in space we perceive a sound source (ventriloquism)
- but can also change what we hear: McGurk effect
McGurk effect
- what is perceived does not match either unisensory
- MSI of auditory + visual
eating as a multisensory experience
- gustation
- olfaction
- vision
- somatosensory (e.g. texture)
- pain
- sound
sound affects of eating
- food tastes crunchier and fresher when the sound is amplified or higher frequencies increased
- white noise can diminish ratings of taste intensity as you can hear the sounds of you eating (airline food?)
