Hearing Speech in Noise: Seeing a Loud Talker is Better

Kim, Jeesun; Sironic, Amanda; Davis, Chris

doi:10.1068/p6941

Cited by 25 publications

(19 citation statements)

References 21 publications

(17 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, talkers adapt visual elements of speech produced in noise such that articulated movements are larger and more correlated with speech acoustics, resulting in an increased perceptual benefit from visible speech (Kim et al, 2011). These adaptations occur automatically in noise but are also part of explicit communicative strategies (Garnier et al, 2010; Hazan and Baker, 2011).…”

Section: Discussionmentioning

confidence: 99%

Perception drives production across sensory modalities: A network for sensorimotor integration of visual speech

et al. 2016

View full text Add to dashboard Cite

Sensory information is critical for movement control, both for defining the targets of actions and providing feedback during planning or ongoing movements. This holds for speech motor control as well, where both auditory and somatosensory information have been shown to play a key role. Recent clinical research demonstrates that individuals with severe speech production deficits can show a dramatic improvement in fluency during online mimicking of an audiovisual speech signal suggesting the existence of a visuomotor pathway for speech motor control. Here we used fMRI in healthy individuals to identify this new visuomotor circuit for speech production. Participants were asked to perceive and covertly rehearse nonsense syllable sequences presented auditorily, visually, or audiovisually. The motor act of rehearsal, which is prima facie the same whether or not it is cued with a visible talker, produced different patterns of sensorimotor activation when cued by visual or audiovisual speech (relative to auditory speech). In particular, a network of brain regions including the left posterior middle temporal gyrus and several frontoparietal sensorimotor areas activated more strongly during rehearsal cued by a visible talker versus rehearsal cued by auditory speech alone. Some of these brain regions responded exclusively to rehearsal cued by visual or audiovisual speech. This result has significant implications for models of speech motor control, for the treatment of speech output disorders, and for models of the role of speech gesture imitation in development.

show abstract

Section: Discussionmentioning

confidence: 99%

Perception drives production across sensory modalities: A network for sensorimotor integration of visual speech

et al. 2016

View full text Add to dashboard Cite

show abstract

“…Lombard speech is one of a suite of acoustic and phonetic modification techniques; these modifications include increases in fundamental frequency (F0) and speech levels, shifted first and second frequencies, increases in higher-frequency energy, and a flattened spectral tilt. At the phonetic level, Lombard speech is characterised by increased vowel duration as well as energy shifts among different classes of phonemes [55][56][57][58][59][60][61][62][63][64][65][66][67][68]. Visually, Lombard speech is characterised by greater global movement of the jaw as well as lip spreading, opening, closing, and protrusion [69,70].…”

Section: Discussionmentioning

confidence: 99%

The impact of automatic exaggeration of the visual articulatory features of a talker on the intelligibility of spectrally distorted speech

Alghamdi

Maddock

Barker

et al. 2017

Speech Communication

View full text Add to dashboard Cite

Visual speech information plays a key role in supporting speech perception, especially when acoustic features are distorted or inaccessible. Recent research suggests that for spectrally distorted speech, the use of visual speech in auditory training improves not only subjects' audiovisual speech recognition, but also their subsequent auditory-only speech recognition. Visual speech cues, however, can be affected by a number of facial visual signals that vary across talkers, such as lip emphasis and speaking style. In a previous study, we enhanced the visual speech videos used in perception training by automatically tracking and colouring a talker's lips. This improved the subjects' audiovisual and subsequent auditory speech recognition compared with those who were trained via unmodified videos or audio-only methods. In this paper, we report on two issues related to automatic exaggeration of the movement of the lips/ mouth area. First, we investigate subjects' ability to adapt to the conflict between the articulation energy in the visual signals and the vocal effort in the acoustic signals (since the acoustic signals remained unexaggerated). Second, we have examined whether or not this visual exaggeration can improve the subjects' performance of auditory and audiovisual speech recognition when used in perception training. To test this concept, we used spectrally distorted speech to train groups of listeners using four different training regimes: (1) audio only, (2) audiovisual, (3) audiovisual visually exaggerated, and (4) audiovisual visually exaggerated and lip-coloured. We used spectrally distorted speech (cochlear-implant-simulated speech) because the longer-term aim of our work is to employ these concepts in a training system for cochlear-implant (CI) users. The results suggest that after exposure to visually exaggerated speech, listeners had the ability to adapt alongside the conflicting audiovisual signals. In addition, subjects trained with enhanced visual cues (regimes 3 and 4) achieved better audiovisual recognition for a number of phoneme classes than those who were trained with unmodified visual speech (regime 2). There was no evidence of an improvement in the subsequent audio-only listening skills, however. The subjects' adaptation to the conflicting audiovisual signals may have slowed down auditory perceptual learning, and impeded the ability of the visual speech to improve the training gains. 1 Highlights. 1. An automatic method that exaggerates the visual articulatory features of talkers in audiovisual speech is presented. 2. An audiovisual conflict after-effect was found, but it becomes negligible after exposing subjects to exaggerated speech. 3. Exaggeration of the visual speech improves the audiovisual recognition of a number of phoneme classes. 4. Audiovisual recalibration to visually exaggerated speech may have impeded learning when used in the audiovisual training.

show abstract

“…Only relatively recently has it been demonstrated that seeing the exaggerated visible articulation of speech produced in noise (Lombard speech) enables better perception of auditory speech in noise, when compared to the visible articulation of quite speech (Kim, Sironic & Davis, 2011). The Alghamdi et al's paper (this issue) investigated the impact of artificially exaggerating the visual articulatory features (the mouth kinematics) on the benefit to speech recognition performance of seeing the talker.…”

Section: Auditory Visual Speech Perceptionmentioning

confidence: 99%

Introduction to the special issue on auditory-visual expressive speech and gesture in humans and machines

Kim

Bailly

Davis

2018

Speech Communication

Self Cite

View full text Add to dashboard Cite

We speak to express ourselves. Sometimes words can capture what we mean; sometimes we mean more than can be said. This is where our visible gestures those dynamic oscillations of our gaze, face, head, hand, arms and bodies-help. Not only do these co verbal visual signals help express our intentions, attitudes and emotion, they also help us engage with our conversational partners to get our message across. Understanding how and when a message is supplemented, shaped and changed by auditory and visual signals is crucial for a science ultimately interesting in the correct interpretation of transmitted meaning. This special issue highlights research articles that explore co verbal and nonverbal signals, a key topic in speech communication since these are crucial ingredients in the interpretation of meaning. That is, the meaning of speech is calibrated, augmented and even changed by co verbal/speech behaviours and gestures including the talker's facial expression, eye contact, gaze direction, arm movements, hand gestures, body motion and orientation, posture, proximity, physical contact, and so on. Understanding expressive signals is a vital step for developing machines that can properly decipher intention and engage as social agents. The special issue is divided into three parts: Auditory visual speech perception; Characterization and perception of auditory visual prosody; Computer generated auditory visual speech. Below, we introduce these papers with a brief review of relevant issues and previous studies, when needed.

show abstract

Hearing Speech in Noise: Seeing a Loud Talker is Better

Cited by 25 publications

References 21 publications

Perception drives production across sensory modalities: A network for sensorimotor integration of visual speech

Perception drives production across sensory modalities: A network for sensorimotor integration of visual speech

The impact of automatic exaggeration of the visual articulatory features of a talker on the intelligibility of spectrally distorted speech

Introduction to the special issue on auditory-visual expressive speech and gesture in humans and machines

Contact Info

Product

Resources

About