How Are Audiovisual Simultaneity Judgments Affected by Multisensory Complexity and Speech Specificity?

Abstract

While stimulus complexity is known to affect the width of the temporal integration window (TIW), a quantitative evaluation of ecologically highly valid stimuli has not been conducted. We assumed that the degree of complexity is determined by the obviousness of the correspondence between the auditory onset and visual movement, and we evaluated the audiovisual complexity using video clips of a piano, a shakuhachi flute and human speech. In Experiment 1, a simultaneity judgment task was conducted using these three types of stimuli. The results showed that the width of TIW was wider for speech, compared with the shakuhachi and piano. Regression analysis revealed that the width of the TIW depended on the degree of complexity. In the second experiment, we investigated whether or not speech-specific factors affected the temporal integration. We used stimuli that either contained natural-speech sounds or white noise. The results revealed that the width of the TIW was wider for natural sentences, compared with white noise. Taken together, the width of the TIW might be affected by both the complexity and speech specificity.

Spatiotemporal models for multisensory integration

“…Classic experiments on the temporal determinants of audiovisual integration usually manipulate the lag between the senses and assess the perception of synchrony, temporal order, and audiovisual speech integration (as measured with the McGurk illusion) through psychophysical forced-choice tasks 17,18 . Among them, we obtained both the audiovisual footage and the psychophysical data from 36 different experiments that used ecological audiovisual stimuli (i.e., speech and performing musicians, Figure 2A, Table 1, for the inclusion criteria, see Supplemental Information): 24 experiments were simultaneity judgments 7,[19][20][21][22][23][24][25][26] , 6 temporal order judgments 24,27 , 6 others assessed the McGurk effect 19,21,27 . All used the same manipulation of audiovisual lags (i.e., method of constant stimuli).…”