Prediction and constraint in audiovisual speech perception

Abstract

During face-to-face conversational speech listeners must efficiently process a rapid and complex stream of multisensory information. Visual speech can serve as a critical complement to auditory information because it provides cues to both the timing of the incoming acoustic signal (the amplitude envelope, influencing attention and perceptual sensitivity) and its content (place and manner of articulation, constraining lexical selection). Here we review behavioral and neurophysiological evidence regarding listeners' use of visual speech information. Multisensory integration of audiovisual speech cues improves recognition accuracy, particularly for speech in noise. Even when speech is intelligible based solely on auditory information, adding visual information may reduce the cognitive demands placed on listeners through increasing precision of prediction. Electrophysiological studies demonstrate oscillatory cortical entrainment to speech in auditory cortex is enhanced when visual speech is present, increasing sensitivity to important acoustic cues. Neuroimaging studies also suggest increased activity in auditory cortex when congruent visual information is available, but additionally emphasize the involvement of heteromodal regions of posterior superior temporal sulcus as playing a role in integrative processing. We interpret these findings in a framework of temporally-focused lexical competition in which visual speech information affects auditory processing to increase sensitivity to auditory information through an early integration mechanism, and a late integration stage that incorporates specific information about a speaker's articulators to constrain the number of possible candidates in a spoken utterance. Ultimately it is words compatible with both auditory and visual information that most strongly determine successful speech perception during everyday listening. Thus, audiovisual speech perception is accomplished through multiple stages of integration, supported by distinct neuroanatomical mechanisms.

Oscillatory phase shapes syllable perception

“…Much research has focused on the role of oscillations in systematically increasing and decreasing the excitability levels of neuronal populations (23,26,27). In this line of reasoning, speech processing is enhanced by aligning the most excitable phase of an oscillation to the incoming speech signal (5,6). Intuitively, our results seem in contrast to this idea, as it appears that neuronal populations coding for separate syllables have phase-specific responses.…”

The motor way: Clinical implications of understanding and shaping actions with the motor system in autism and drug addiction

“…Even if it goes beyond the aims of this article, a further and more basic level of social cognition can be drawn-that is, primary social cognition (Jessen & Grossmann, 2014;Whalen et al, 2004). This level concerns the pre-cognitive and pre-motor understanding of elementary social information that is processed by-for example-visual and auditory perceptual processes or attentional mechanisms (for recent reviews, see Peelle & Sommers, 2015;Petersen & Posner, 2012;Vuilleumier, 2015). For example, biological motion processing is an emblematic hallmark of primary social cognition.…”

Visual Context Enhanced: The Joint Contribution of Iconic Gestures and Visible Speech to Degraded Speech Comprehension

“…Yet, these lip movements are inherently part of natural, face-to-face communication: Lip movements can provide temporal information about the speech signal (e.g., on the amplitude envelope) and information on the spatial location of a speaker's articulators (e.g., place and manner of articulation), which can be specifically useful when perceiving speech in adverse listening conditions. In addition, lip movements can convey phonological information, because of the form-form relationship between lip movements and syllables or segments that are present in the speech stream (for a recent review, see Peelle & Sommers, 2015).…”