Visual Speech Benefit in Clear and Degraded Speech Depends on the Auditory Intelligibility of the Talker and the Number of Background Talkers

Blackburn, Catherine L; Kitterick, Pádraig T.; Jones, Graeme A.; Sumner, Christian J.; Stacey, Paula C.

doi:10.1177/2331216519837866

Cited by 8 publications

(5 citation statements)

References 37 publications

(55 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Within the TRF framework, we provide first evidence that lip movements enhance acoustic-controlled neural speech tracking (Figure 2C). A general enhancement was observed for both single-and multi speaker speech, which is in line with behavioral findings that visual speech features enhance intelligibility under clear speech conditions as well (Blackburn et al, 2019;Stacey et al, 2016).…”

Section: Benefit Of Lip Movementssupporting

confidence: 88%

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

Reisinger

Gillis

Suess

et al. 2023

Preprint

View full text Add to dashboard Cite

Observing lip movements of a speaker is known to facilitate speech understanding, especially in challenging listening situations. Converging evidence from neuroscientific studies shows enhanced processing of audiovisual stimuli. However, the interindividual variability of this visual benefit and its consequences on behavior are unknown. Here, we analyzed source-localized magnetoencephalographic (MEG) responses from normal-hearing participants listening to audiovisual speech with or without an additional distractor speaker. Using temporal response functions (TRFs), we show that neural responses to lip movements are, in general, enhanced when speech is challenging. After conducting a crucial control for speech acoustics, we show that lip movements effectively contribute to higher neural speech tracking, particularly when a distractor speaker is present. However, the extent of this visual benefit varied greatly among participants. Probing the behavioral relevance, we show that individuals who benefit more from lip movement information in terms of neural speech tracking, show a stronger drop in performance and an increase in perceived difficulty when the mouth is occluded by a surgical face mask. By contrast, no effect was found when the mouth was not occluded. We provide novel insights on how the benefit of lip movements in terms of neural speech tracking varies among individuals. Furthermore, we reveal its behavioral relevance by demonstrating negative consequences for behavior when visual speech is absent. Our results also offer potential implications for future objective assessments of audiovisual speech perception.

show abstract

Section: Benefit Of Lip Movementssupporting

confidence: 88%

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

Reisinger

Gillis

Suess

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…These main outcomes of the study have several implications, both for basic science as well as in terms of their potential practical applications. Intriguingly, we show here that the improvement of speech comprehension through audio-tactile training was in the order of magnitude (~ 10 dB in SRT) similar or higher than that found for audio-visual speech-in-noise testing settings, when the performance is compared to only auditory [5,48,49,50,51]. The reported improvement in SRT was by 3 to 11dB, although the applied methodology and the language content varied.…”

Section: Discussionsupporting

confidence: 69%

Speech-to-touch sensory substitution: a 10-decibel improvement in speech-in-noise understanding after a short training

Cieśla

Wolak

Lorens

et al. 2021

Preprint

View full text Add to dashboard Cite

Understanding speech in background noise is challenging. Wearing face-masks during COVID19-pandemics made it even harder. We developed a multi-sensory setup, including a sensory substitution device (SSD) that can deliver speech simultaneously through audition and as vibrations on fingertips. After a short training session, participants significantly improved (16 out of 17) in speech-in-noise understanding, when added vibrations corresponded to low-frequencies extracted from the sentence. The level of understanding was maintained after training, when the loudness of the background noise doubled (mean group improvement of ~ 10 decibels). This result indicates that our solution can be very useful for the hearing-impaired patients. Even more interestingly, the improvement was transferred to a post-training situation when the touch input was removed, showing that we can apply the setup for auditory rehabilitation in cochlear implant-users. Future wearable implementations of our SSD can also be used in real-life situations, when talking on the phone or learning a foreign language. We discuss the basic science implications of our findings, such as we show that even in adulthood a new pairing can be established between a neuronal computation (speech processing) and an atypical sensory modality (tactile). Speech is indeed a multisensory signal, but learned from birth in an audio-visual context. Interestingly, adding lip reading cues to speech in noise provides benefit of the same or lower magnitude as we report here for adding touch.

show abstract

“…perception in noise, vocoded speech, or when listening to an unfamiliar language, see e.g. Banks et al, 2021 ; Blackburn et al, 2019 ; Lusk & Mitchel, 2016 ; Worster et al, 2018 ).…”

Section: Discussionmentioning

confidence: 99%

How do face masks impact communication amongst deaf/HoH people?

et al. 2022

View full text Add to dashboard Cite

Face coverings have been key in reducing the spread of COVID-19. At the same time, they have hindered interpersonal communication, particularly for those who rely on speechreading to aid communication. The available research indicated that deaf/hard of hearing (HoH) people experienced great difficulty communicating with people wearing masks and negative effects on wellbeing. Here we extended these findings by exploring which factors predict deaf/HoH people’s communication difficulties, loss of information, and wellbeing. We also explored the factors predicting perceived usefulness of transparent face coverings and alternative ways of communicating. We report the findings from an accessible survey study, released in two written and three signed languages. Responses from 395 deaf/HoH UK and Spanish residents were collected online at a time when masks were mandatory. We investigated whether onset and level of deafness, knowledge of sign language, speechreading fluency, and country of residence predicted communication difficulties, wellbeing, and degree to which transparent face coverings were considered useful. Overall, deaf/HoH people and their relatives used masks most of the time despite greater communication difficulties. Late-onset deaf people were the group that experienced more difficulties in communication, and also reported lower wellbeing. However, both early- and late-onset deaf people reported missing more information and feeling more disconnected from society than HoH people. Finally, signers valued transparent face shields more positively than non-signers. The latter suggests that, while seeing the lips is positive to everyone, signers appreciate seeing the whole facial expression. Importantly, our data also revealed the importance of visual communication other than speechreading to facilitate face-to-face interactions.

show abstract

Visual Speech Benefit in Clear and Degraded Speech Depends on the Auditory Intelligibility of the Talker and the Number of Background Talkers

Cited by 8 publications

References 37 publications

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

Speech-to-touch sensory substitution: a 10-decibel improvement in speech-in-noise understanding after a short training

How do face masks impact communication amongst deaf/HoH people?

Contact Info

Product

Resources

About