Human trimodal perception follows optimal statistical inference

Abstract

Our nervous system typically processes signals from multiple sensory modalities at any given moment and is therefore posed with two important problems: which of the signals are caused by a common event, and how to combine those signals. We investigated human perception in the presence of auditory, visual, and tactile stimulation in a numerosity judgment task. Observers were presented with stimuli in one, two, or three modalities simultaneously and were asked to report their percepts in each modality. The degree of congruency between the modalities varied across trials. For example, a single flash was paired in some trials with two beeps and two taps. Cross-modal illusions were observed in most conditions in which there was incongruence among the two or three stimuli, revealing robust interactions among the three modalities in all directions. The observers' bimodal and trimodal percepts were remarkably consistent with a Bayes-optimal strategy of combining the evidence in each modality with the prior probability of the events. These findings provide evidence that the combination of sensory information among three modalities follows optimal statistical inference for the entire spectrum of conditions.

Counting visual and tactile events: The effect of attention on multisensory integration

“…This is consistent with data from most previous studies, all of which reported insignificant differences (Andersen et al, 2004;Violentyev et al, 2005;Wozny et al, 2008). It should be noted that Philippi et al (2008)-who studied uni-, bi-, and trimodal temporal-numerosity judgments for a wide range of ISIs (20-320 msec) and number of events ( 2-10)-did find significant differences in bias for uniand multimodal conditions; however, they did not report significant differences for the vision and touch combination at the ISIs that were tested here.…”

“…In the no-cue condition, the monitor displayed only an exclamation mark, followed by the stimulus presentation. Directly after the presentation of the stimulus, the fixation cross was replaced ing et al, 2007;Roach, Heron, & McGraw, 2006;Sato, Toyoizumi, & Aihara, 2007;Shams, Ma, & Beierholm, 2005;Wozny, Beierholm, & Shams, 2008), however, have concluded that sensory integration is partial rather than complete. That is, the sum of the weights for the modalities to be integrated is less than one, and the information of each modality remains individually accessible when all are combined.…”

“…In cued conditions, however, the strength of the integration and the ability to ignore a channel are confounded. Others (Körding et al, 2007;Wozny et al, 2008) have studied conditions in which participants reported on all the modalities that were involved and hence were not cued to ignore any. Because these experiments were designed for other purposes, they did not compare these conditions with cued conditions.…”

Optimal integration of auditory and vibrotactile information for judgments of temporal order.

“…More recently, studies have begun to assess judgments which may depend partially upon temporal acuity, such as counting the number of rapid transient events in a short train (Bresciani, Dammeier & Ernst, 2006;Bresciani & Ernst, 2007;Shams et al, 2005;Wozny, Beierholm & Shams, 2008) or comparing the rate of a flickering or fluttering stimulus (Roach, Heron & McGraw, 2006;Wada, Kitagawa & Noguchi, 2003 In TOJ tasks, subjects are presented with two brief stimuli in rapid temporal succession and asked to discriminate which came first. The method has been widely used throughout the history of experimental psychology (Spence, Shore & Klein, 2001).…”

Phantom percepts: Tinnitus and pain as persisting aversive memory networks

“…Incongruence, Pain, and Tinnitus Multisensory mechanisms involving crossmodal congruency are involved in bodily self-consciousness (66-68), and crossmodal illusions are observed in most conditions in which there is incongruence among two or three stimuli of different modalities (auditory, visual, somatosensory) (69). Whereas synchronous stimulation of external objects, such as a rubber hand, can lead to the illusion of body ownership (70), asynchronous stimulation in one sensory modality or between the senses can lead to abnormal sensory perception (71,72).…”