Multisensory integration

Question 1

What is Moravec’s paradox?

Answer 1

“It is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility.”
For computers:
High level cognitive reasoning- relatively easy
Low level (perception, action)- suprisingly hard

Question 2

What is an example of Moravecs paradox?

Answer 2

• First time a computer beat the worlds best chess player
  (‘hard’ task)
• robots bad at walking (‘easy’ task)

Question 3

What happend when you anesthetize the fingers?

Answer 3

• blocks all touch sensations from the fingers
• Does not affect motor control! (brain)
• has an effect on lighting a match- makes her clumsy
• we need multisensory integration- performance can be changed

Question 4

What is multisensory perception?

Answer 4

• integrating senses to get a coherent percept that tells us something about an object

Question 5

What are the challenges in multisensory integration?

Answer 5

1. How to transform representations from different senses into a common representational space? (different sensory info from different senses come in different formats)
2. How to integrate information from different senses into a coherent percept?

Question 6

What body parts are sensory information are located in what body part?

Answer 6

• Vision → eye-centred/retinal
  location of visual stimulus on the retina
• Audition → head-centred
  location of sound source with respect to the ears
• Touch → body-centred
  location of tactile stimulus on skin
• need to convert between these reference frames and to external space (world coordinates)

Question 7

What is an example of reference frames? (google snake)

Answer 7

Plyer perspective: coordinates of snake in the world, top down view
Snake perspective: sees world through its eyes, can turn left or right
Problem: sensory input in player perspective, controls in snake perspective
- need to be able to convert between reference frames from ours to the snakes

Question 8

What is a human example of coordinate transformations?

Answer 8

• human looking down at dog, dog at centre of vision
• head centred reference frame (dog would be at the centre of that), same with eye centrerd frame
  If dog moves to the side: dog is still at the centre of eye frame because we turn our eyes but from our head-centred reference frame.
• Haven’t moved our head so the dog will not be at the centre of our reference frame (haven’t turned head)
• If dog moves back to centre but we don’t move our eyes: dog is not at centre of eye reference frame but it is into the middle of the head centred reference frame

Question 9

What do we need to know to change reference frames?

Answer 9

• Eye-to-head: need to know orientation of eyes
• Head-to-body: need to know orientation of head
  → Converting between reference frames requires us to know the position and orientation of body parts.
• in other words we need a body schema

Question 10

What is the body schema?

Answer 10

Haggard and Wolpert 2005:
‘‘a representation of the positions of body parts in space, which is updated during body movement”

Question 11

What are 7 propterties of the body schema? (Haggard and Wolpert)

Answer 11

1. Spatially coded: position of each body part in external space
2. Modular: different body parts processed separately
3. Updated with movement: automated and “always on”
4. Adaptable: Changes when the body changes
5. Supramodal: Combines input from multiple senses, e.g. proprioception, touch, vision, audition
6. Coherent: Spatial continuity when resolving perceptual conflicts
7. Interpersonal: observed actions are represented within the same body schema

Question 11

What are the differences between body schema and body image?

Answer 11

Body schema:
- sensorimotor representation that guides action
- does not require awareness
Body image:
- how do you imagine your body
- how do you feel about your body
- conscious (mostly visual) representation.

Question 12

What are body illusions?

Answer 12

• can teach a lot about how the body schema work
• body deviating from reality

Question 13

What is the pinocchio illusion?

Answer 13

1) Vibrating the biceps will make arm feel extended (whilst holding nose)
2) If arm is extended, but we are grasping our nose:
-> nose must be really long

Question 14

What is the rubber hand illusion?

Answer 14

Setup:
- Participant’s real hand hidden from view
- Experimenter strokes real and rubber hand simultaneously
- Participant sees and FEELS rubber hand being stroked
- Participant starts to feel as if rubber hand is real hand
-> Vision and touch: multisensory integration (seeing rubber hand and feeling on own)
(for 25% of people this doesn’t work at all- lots of differences in how people feel this)

Question 15

Does body posture affect perception? What task can we use to test this?

Answer 15

Temporal order judgement task:
- Setup: stimulate both hands in random order
- Task: Indicate which hand was stimulated first
- Conditions: Arms uncrossed/crossed
Results:
- psychophysical curve shows when hands were uncrossed they could easily tell (if the pause was long enough), if arms were crossed marticipants often mix up which hand was stimulated, more confusion with short delays
- Therefore body schema affecting basic perception

Question 16

How does the body schema develop?

Answer 16

• At 6 months old, the body schema starts to interfere with tactile orienting
• crossed or uncrossed feet at 4m and 6m, at 4m theres no difference in performance at reaching towards the correct foot, at 6m they perform better if the feet are uncrossed (this is the same percent correct as at 4m) but performance is worse when feet are crossed (chance levels)

Question 17

What is the cross-modal integration task?

Answer 17

• Participants receive a tactile stimulus and respond with the location (up or down).
• Visual distractor (up or down) is presented nearby on the same hand or the other hand.
• record reaction time
• difference in effect size depending on where the distractors show up, effects RT
• A Congruency effect exists for both hands – evidence for cross-modal integration and visual interference.
• But: effect greater for same hand.

Question 18

Is the greater inference for congruency effect based on hand location (body schema) or visual field?

Answer 18

• Arm-crossing
• Changing the mapping of the body schema
• tactile stimulus on same side of the body
• visual stimulus on different side of body
• Effect of visual distractor moves with the hand during arm-crossing
• The body schema (not visual representation of external space), mediates the integration between vision and tactile stimulation.
• Cross-modal interactions are mediated by body schema.

Question 19

What are some disorders of body schema? Autopagnosia

Answer 19

• Inability to locate body parts
• Loss of spatial unity of body
• Patients can name body parts but relative order lost
• Finger agnosia: Fused percept of fingers

Question 20

What are some disorders of body schema? Alice in Wonderland syndrome

Answer 20

Alice in Wonderland syndrome:
- Distortion in size perception
- Body parts might appear smaller (microsomatognosia) or larger than their true size (macrosomatognosia)
- Can affect whole body
- Associated with childhood and migraines

Question 21

What are some disorders of body schema? Phantom limbs

Answer 21

• After limb loss, presence of limb is still felt
• Can include agency over limb (movement)
• Often associated with pain
• Phantom limb can change size over time (e.g. shrink: telescoping)

Question 22

What is meant by peripersonal space?

Answer 22

• The space immediately surrounding our bodies
• Objects within peripersonal space can be grasped and manipulated immediately
• Extent not fixed: can contract and expand

Question 23

How do neurons incorporate tools?

Answer 23

• neurons respond when visual objects move close to hand into peripersonal space
• response space expands during tool use
• expansion of peripersonal space during tool use reflected in neural responses

Question 24

How are tool use and body schema related?

Answer 24

- tools are incorporated into the body schema during use - Cross-modal congruency effects apply during tool use - No crossing of body parts, only tools are crossed - Same delay effects as seen during arm crossing → Tools become part of body schema

Question 25

What do arm-crossing experiments tell us about the body schema?

Answer 25

That the body schema can affect perception