Lipreading: A Review of Its Continuing Importance for Speech Recognition With an Acquired Hearing Loss and Possibilities for Effective Training

Abstract

Purpose: The goal of this review article is to reinvigorate interest in lipreading and lipreading training for adults with acquired hearing loss. Most adults benefit from being able to see the talker when speech is degraded; however, the effect size is related to their lipreading ability, which is typically poor in adults who have experienced normal hearing through most of their lives. Lipreading training has been viewed as a possible avenue for rehabilitation of adults with an acquired hearing loss, but most training approaches have not been particularly successful. Here, we describe lipreading and theoretically motivated approaches to its training, as well as examples of successful training paradigms. We discuss some extensions to auditory-only (AO) and audiovisual (AV) speech recognition. Method: Visual speech perception and word recognition are described. Traditional and contemporary views of training and perceptual learning are outlined. We focus on the roles of external and internal feedback and the training task in perceptual learning, and we describe results of lipreading training experiments. Results: Lipreading is commonly characterized as limited to viseme perception. However, evidence demonstrates subvisemic perception of visual phonetic information. Lipreading words also relies on lexical constraints, not unlike auditory spoken word recognition. Lipreading has been shown to be difficult to improve through training, but under specific feedback and task conditions, training can be successful, and learning can generalize to untrained materials, including AV sentence stimuli in noise. The results on lipreading have implications for AO and AV training and for use of acoustically processed speech in face-to-face communication. Conclusion: Given its importance for speech recognition with a hearing loss, we suggest that the research and clinical communities integrate lipreading in their efforts to improve speech recognition in adults with acquired hearing loss.

Synthetic faces generated with the facial action coding system or deep neural networks improve speech-in-noise perception, but not as much as real faces

“…Another experimental approach is to present visual-only speech without any auditory input, although this condition differs from most real-world situations with the exception of profound deafness. The ability to extract information about speech from the face of the talker (known as lipreading or speechreading) varies widely across individuals, and it may be possible to improve this ability through training ( Auer and Bernstein, 2007 ; Bernstein et al, 2022 ).…”

Assessing Visual Speech Communication Abilities: Cross-situational Consistency in Lipreadability

“…Impaired hearing during this period puts a premium on attention to visual speech information and leads to development of greater lipreading ability. Beginning around age 6, however, improving visual speech recognition appears to be much more difficult to achieve than the incidental learning that occurs early on (for a review, see Bernstein et al, 2022).…”

Hearing loss, depression, and cognition in younger and older adult CI candidates

“…In our clinical experience, many patients with severe and profound hearing loss and cochlear implant candidates can read speech because they rely on visual perception (viseme) to follow a conversation. Studies show that a combination of visual and auditory speech information improves speech recognition [without hearing loss ( 142 – 144 )]. Our hypothesis is that reading speech also trains visual memory.…”