Auditory-Visual Speech Perception and Auditory-Visual Enhancement in Normal-Hearing Younger and Older Adults

Sommers, Mitchell S.; Tye‐Murray, Nancy; Spehar, Brent

doi:10.1097/00003446-200506000-00003

Cited by 220 publications

(330 citation statements)

References 24 publications

Supporting

Mentioning

291

Contrasting

Unclassified

Order By: Relevance

“…One potential shortcoming of previous in measuring integration relates to the inherent difficulty of defining what is meant by "integration." For instance, integration has been operationally defined as susceptibility to the McGurk effect (Grant, 2002), audiovisual enhancement in accuracy scores (Bergeson & Pisoni, 2004;Sommers et al, 2005;Sumby & Pollack, 1954), superadditivity in the neural signal (Calvert & Campbell, 2003;Calvert, Campbell, & Brammer, 2000), the combining of the intermodality information (as in FLMP; Massaro, 2004), or the difference between obtained accuracy scores compared to model predictions derived from confusion matrices (Grant, 2002;Grant et al 1998). A promising course of action has been to formalize these definitions and use them to test models against one another (e.g., Grant et al, 1998;Massaro, 1987aMassaro, , 1998.…”

Section: Discussionmentioning

confidence: 99%

Section: Measures Of Capacity and Relative Efficiencymentioning

confidence: 99%

“…Importantly, it is possible that different listeners will use differing strategies. Such variation may depend on hearing ability, cognitive status, and susceptibility to the McGurk effect; all of which are known to influence multisensory integration abilities (e.g., Altieri & Hudock, 2014;Erber, 2003;Sommers, Tye-Murray, & Spehar, 2005).First, capacity is assessed by comparing the empirical RT distributions from trials in which both auditory and visual information are presented to the RT distributions obtained from trials in which auditory and visual-only information is present. This is done using integrated hazard functions, as follows.…”

mentioning

confidence: 99%

See 2 more Smart Citations

The McGurk effect: An investigation of attentional capacity employing response times

Altieri

Lentz

Townsend

et al. 2016

Atten Percept Psychophys

View full text Add to dashboard Cite

This paper proposes a novel approach to assess audiovisual integration for both congruent and incongruent speech stimuli using reaction times (RT). The experiments are based on the McGurk effect, in which a listener is presented with incongruent audiovisual speech signals. A typical example involves the auditory consonant/b/combined with a visually articulated/g/, often yielding a perception of/d/. We quantify the amount of integration relative to the predictions of a parallel independent model as a function of attention and congruency between auditory and visual signals. We assessed RT distributions for congruent and incongruent auditory and visual signals in a within-subjects signal detection paradigm under conditions of divided versus focused attention. Results showed that listeners often received only minimal benefit from congruent auditory visual stimuli, even when such information could have improved performance. Incongruent stimuli adversely affected performance in divided and focused attention conditions. Our findings support a parallel model of auditory-visual integration with interactions between auditory and visual channels.Keywords Speech perception . Attention: Selective . Reaction time methodsWhat cognitive mechanisms underlie speech recognition when audition is supplemented with visual information? The modern era of research into how auditory and visual speech cues interact began with Sumby and Pollack's (1954) seminal experimental work on audiovisual (AV) enhancement: They showed that visual cues provided by a talker's lip-movements facilitate auditory recognition across a range of signal-to-noise ratios. However, Massaro (1987a) proved that this outcome did not necessarily demonstrate integration since a singlechannel model could theoretically predict the results.In a critical study two decades later, McGurk and Macdonald (1976) reported a dramatic perceptual integration phenomenon that resulted from the presentation of incongruent auditory-visual speech signals. In what became known as the "McGurk effect," presentation of the auditory consonant/ b/over a visually articulated/g/yielded a fused percept of/d/. Audiovisual fusions such as these occur when the perceptual system maps cues from conflicting signals onto a phonemic category distinct from either input signal.1 Thus, the McGurk effect is a prime candidate with which to probe the mechanisms underlying integration.Several studies of the McGurk effect have been carried out, with the majority using mean accuracy as the dependent variable. In these studies, performance in auditory and visualonly trials is compared to accuracy in audiovisual trials, usually via confusion matrices (e.g., Massaro, 1987aMassaro, , 1998Massaro, , 2004. These experimental designs and modeling efforts have shed considerable light on speech integration: Auditory and visual cues appear to interact in a multiplicative manner 1 This cannot be demonstrated conclusively without also measuring responses to the single modality presentations (cf. Massaro 1987a(cf. M...

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Measures Of Capacity and Relative Efficiencymentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

The McGurk effect: An investigation of attentional capacity employing response times

Altieri

Lentz

Townsend

et al. 2016

Atten Percept Psychophys

View full text Add to dashboard Cite

show abstract

“…Picou et al (2011) found that the person with low WMC did not derive any benefit from the presence of visual cues whereas person with high WMC did derive benefit from the presence of visual cues in cued recall of words. Sommers, Tye-Murray and Spehar (2005) found that the AV integration for speech perception in noise was similar across young and older adults with normal hearing, but the young adults had better speech reading skills compared to older adults and hence had better performance in the AV modality.…”

Section: Visual Cuesmentioning

confidence: 90%

Exploring Cognitive Spare Capacity : Executive Processing of Degraded Speech

Mishra¹

2014

View full text Add to dashboard Cite

Cognitive resources, specifically working memory capacity are used for listening to speech, especially in noise. Cognitive resources are limited, and if listeners allocate a greater share of these resources to recovering the input signal in noise, fewer resources are available for interpreting and encoding its linguistic content. Although the importance of CSC for individual success in communicative situations has been acknowledged, this concept has not hitherto been explored experimentally. In this thesis, a CSC test (CSCT) was developed and administered to young adults with normal hearing and older adults with age-related hearing loss. CSCT required executive processing of speech at different memory loads with and without visual cues in different noise conditions. A free recall task using the same material was administered for comparison purposes and a battery of cognitive tests was administered to understand the relation between CSC and established cognitive concepts. The aims of the thesis were to investigate how CSC is influenced by 1) different executive demands and memory loads; 2) background noise; 3) visual cues; 4) aging and concomitant hearing loss. The results showed that 1) CSC was sensitive to memory load, and updating demands reduced CSC more than inhibition demands; 2) CSC was reduced in background noise compared to quiet; 3) visual cues enhanced CSC especially in noise; 4) CSC was reduced with ageing and concomitant hearing loss especially when visual cues were absent, memory demands were increased and background noise was speech-like. The main finding of this thesis was that visual cues enhanced CSC for older individuals with hearing loss, specifically in adverse listening conditions. This demonstrates the importance of audiovisual testing in audiological assessment. Further, specific cognitive resources depleted during listening in noise were at least partially compensated by other cognitive functions. This thesis is the first step towards a theoretical understanding of CSC and in future, tests of CSC may play a crucial role in planning rehabilitation of persons with hearing loss.

show abstract

“…HL results primarily in a decline in auditory speech perception (Humes & Roberts 1990); however, changes in visual perceptual abilities indicate that adaptation to HL is not limited to the auditory modality. On the one hand, younger adults with early-onset hearing impairment are better at lipreading sentences than controls (Auer et al 2007), but even in normal-hearing (NH) people, this ability declines with age (Cienkowski & Carney 2002;Hay-McCutcheon et al 2005;Sommers et al 2005). Although investigations of unimodal perception in isolation help us understand how HL in older adults affects auditory and visual abilities, everyday perception for most people involves the combination of these inputs.…”

Section: Introductionmentioning

confidence: 99%

Audiovisual Deficits in Older Adults with Hearing Loss: Biological Evidence

Musacchia

Arum²,

Nicol³

et al. 2009

Ear &Amp; Hearing

View full text Add to dashboard Cite

Objective:To examine the impact of hearing loss (HL) on audiovisual (AV) processing in the aging population. We hypothesized that agerelated HL would have a pervasive effect on sensory processing, extending beyond the auditory domain. Specifically, we predicted that decreased auditory input to the neural system, in the form of HL over time, would have deleterious effects on multisensory mechanisms. Design:This study compared AV processing between older adults with normal hearing (N ϭ 12) and older adults with mild to moderate sensorineural HL (N ϭ 12). To do this, we recorded cortical evoked potentials that were elicited by watching and listening to recordings of a speaker saying the syllable "bi." Stimuli were presented in three conditions: when hearing the syllable "bi" (auditory), when viewing a person say "bi" (visual), and when seeing and hearing the syllables simultaneously (AV). Presentation level of the auditory stimulus was set to ϩ30 dB SL for each listener to equalize auditory input across groups. Results:In the AV condition, the normal-hearing group showed a clear and consistent decrease in P1 and N1 latencies as well as a reduction in P1 amplitude compared with the sum of the unimodal components (auditory ϩ visual). These integration effects were absent or less consistent in HL participants.Conclusions: Despite controlling for auditory sensation level, visual influence on auditory processing was significantly less pronounced in HL individuals compared with controls, indicating diminished AV integration in this population. These results demonstrate that HL has a deleterious effect on how older adults combine what they see and hear. Although auditory amplification vastly improves the communication abilities in most HLs, the associated atrophy of multisensory mechanisms may contribute to a patient's difficulty in everyday settings. Our findings and related studies emphasize the potential value of multimodal tasks and stimuli in the assessment and rehabilitation of hearing impairments.

show abstract

Auditory-Visual Speech Perception and Auditory-Visual Enhancement in Normal-Hearing Younger and Older Adults

Cited by 220 publications

References 24 publications

The McGurk effect: An investigation of attentional capacity employing response times

The McGurk effect: An investigation of attentional capacity employing response times

Exploring Cognitive Spare Capacity : Executive Processing of Degraded Speech

Audiovisual Deficits in Older Adults with Hearing Loss: Biological Evidence

Contact Info

Product

Resources

About