Perception of Incongruent Audiovisual Speech: Distribution of Modality-Specific Responses

Abstract

Purpose Multimodal sensory integration in audiovisual (AV) speech perception is a naturally occurring phenomenon. Modality-specific responses such as auditory left, auditory right, and visual responses to dichotic incongruent AV speech stimuli help in understanding AV speech processing through each input modality. It is observed that distribution of activity in the frontal motor areas involved in speech production has been shown to correlate with how subjects perceive the same syllable differently or perceive different syllables. This study investigated the distribution of modality-specific responses to dichotic incongruent AV speech stimuli by simultaneously presenting consonant–vowel (CV) syllables with different places of articulation to the participant's left and right ears and visually. Design A dichotic experimental design was adopted. Six stop CV syllables /pa/, /ta/, /ka/, /ba/, /da/, and /ga/ were assembled to create dichotic incongruent AV speech material. Participants included 40 native speakers of Norwegian (20 women, M age = 22.6 years, SD = 2.43 years; 20 men, M age = 23.7 years, SD = 2.08 years). Results Findings of this study showed that, under dichotic listening conditions, velar CV syllables resulted in the highest scores in the respective ears, and this might be explained by stimulus dominance of velar consonants, as shown in previous studies. However, this study, with dichotic auditory stimuli accompanied by an incongruent video segment, demonstrated that the presentation of a visually distinct video segment possibly draws attention to the video segment in some participants, thereby reducing the overall recognition of the dominant syllable. Furthermore, the findings here suggest the possibility of lesser response times to incongruent AV stimuli in females compared with males. Conclusion The identification of the left audio, right audio, and visual segments in dichotic incongruent AV stimuli depends on place of articulation, stimulus dominance, and voice onset time of the CV syllables.

Audiovisual Speech Perception in Diotic and Dichotic Listening Conditions

“…The previous study by Sandhya et al [39] Considering dichotic listening as a task in which inputs to the two ears compete, a concordant visual signal benefits the respective auditory signal competing with the discordant auditory signal. This may be regarded as similar to speech perception in the presence of noise [2,52], where supplementary visual information facilitates speech perception.…”

“…Recently, Sandhya et al [39] investigated the distribution of modality-specific responses to dichotic incongruent AV speech stimuli in normal hearing young adults. The study used incongruent AV stimuli with simultaneous presentation of three different stop CV syllables to the right ear, left ear, and visually.…”

“…These features of speech segments are independent, and are often described using phonological categories [40]. In the case of multimodal audiovisual speech perception, the perceptual role of temporal structure in audiovisual speech needs further research [39,41]. The existing literature does not provide enough evidence about the role of VOT and POA on perception of speech under diotic and dichotic AV conditions.…”