Gated audiovisual speech identification in silence vs. noise: effects on time and accuracy

Abstract

This study investigated the degree to which audiovisual presentation (compared to auditory-only presentation) affected isolation point (IPs, the amount of time required for the correct identification of speech stimuli using a gating paradigm) in silence and noise conditions. The study expanded on the findings of Moradi et al. (under revision), using the same stimuli, but presented in an audiovisual instead of an auditory-only manner. The results showed that noise impeded the identification of consonants and words (i.e., delayed IPs and lowered accuracy), but not the identification of final words in sentences. In comparison with the previous study by Moradi et al., it can be concluded that the provision of visual cues expedited IPs and increased the accuracy of speech stimuli identification in both silence and noise. The implication of the results is discussed in terms of models for speech understanding.

Exploring Cognitive Spare Capacity : Executive Processing of Degraded Speech

“…However, Allen, Baddeley & Hitch (2006) from five behavioural experiments concluded that although the presence of visual cues demands attention similar to unimodal stimuli initially, but AV integration does not require additional attentional resources. Moradi, Lidestam and Rönnberg (2013) found that the AV speech recognition in the presence of noise for young adults with normal hearing is faster, more accurate and less effortful than auditory-only speech recognition, and inferred that AV presentation taxes cognitive resources to a lesser extent by reducing working memory load. Yovel and Belin (2013) suggested that despite sensory differences, the neurocognitive mechanisms engaged by perceiving faces and voices are highly similar, facilitating integration of visual and speech information.…”

Audiovisual presentation of video-recorded stimuli at a high frame rate

“…In order to accomplish this with standard computer equipment, Matlab (2009b, 32-bit) and Psychophysics Toolbox 3 (Brainard, 1997;Pelli, 1997; Kleiner, Brainard, & Pelli, 2007) can be used. In Moradi et al (2013), 120 fps was used. However, the computer's processing speed determines the spatial resolution that can be used for video presentation at the desired rate.…”

“…Methods for accomplishing this comprise of recording audio and video separately using an exact synchronization signal, editing the recordings and finding exact synchronization points, and presenting the synchronized audiovisual stimuli with a desired frame-rate on a CRT display using Matlab and Psychophysics Toolbox 3. The methods from an empirical gating study (Moradi, Lidestam, & Rönnberg, 2013) are presented as an example of implementation of playback at 120 fps.Keywords: psychophysics, frame rate, audiovisual, synchronization, temporal resolution AUDIOVISUAL STIMULI AT HIGH FRAME-RATE 3 …”

“…In film production the synchronization error may therefore be as large as 1 s / 24 ≈ 42 ms. The method presented here can be used to precisely and reliably ascertain synchronization well below 1 ms, and was successfully implemented in Moradi, Lidestam, and Rönnberg (2013). …”

“…Developing and describing such a procedure was however not within the scope of the present paper. Therefore Moradi et al (2013) used a CRT display, with the desired result.Benefit of high frame rate in research on perception and psychophysics AUDIOVISUAL STIMULI AT HIGH FRAME-RATE 6 As of today, playback of video-recordings at frame rates exceeding 30 fps, with or without synchronized sound, is not available by standard consumer technologies (but 48 fps playback is beginning to be used in some cinemas, see HFR, n.d.). Whenever naturalness and smoothness of motion is required in stimulus presentation, fast-enough frame rate is needed.…”