The contribution of dynamic visual cues to audiovisual speech perception

Jaekl, Philip; Pesquita, Ana; Alsius, Agnès; Munhall, Kevin G.; Soto‐Faraco, Salvador

doi:10.1016/j.neuropsychologia.2015.06.025

Cited by 14 publications

(9 citation statements)

References 74 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Research on audiovisual speech perception in aging is also relevant to this review, since the temporal relationships between auditory and visual speech cues ( Chandrasekaran et al, 2009 )facilitate both auditory speech detection ( Grant and Seitz, 2000 ) and recognition ( ten Oever et al, 2013 ; Jaekl et al, 2015 ). Visual facilitation of auditory speech detection is reduced in older adults ( Tye-Murray et al, 2011 ).…”

Section: Age-related Changes In Audiovisual Temporal Perceptionmentioning

confidence: 99%

Audiovisual Temporal Perception in Aging: The Role of Multisensory Integration and Age-Related Sensory Loss

Brooks

Chan

Anderson

et al. 2018

Front. Hum. Neurosci.

View full text Add to dashboard Cite

Within each sensory modality, age-related deficits in temporal perception contribute to the difficulties older adults experience when performing everyday tasks. Since perceptual experience is inherently multisensory, older adults also face the added challenge of appropriately integrating or segregating the auditory and visual cues present in our dynamic environment into coherent representations of distinct objects. As such, many studies have investigated how older adults perform when integrating temporal information across audition and vision. This review covers both direct judgments about temporal information (the sound-induced flash illusion, temporal order, perceived synchrony, and temporal rate discrimination) and judgments regarding stimuli containing temporal information (the audiovisual bounce effect and speech perception). Although an age-related increase in integration has been demonstrated on a variety of tasks, research specifically investigating the ability of older adults to integrate temporal auditory and visual cues has produced disparate results. In this short review, we explore what factors could underlie these divergent findings. We conclude that both task-specific differences and age-related sensory loss play a role in the reported disparity in age-related effects on the integration of auditory and visual temporal information.

show abstract

Section: Age-related Changes In Audiovisual Temporal Perceptionmentioning

confidence: 99%

Audiovisual Temporal Perception in Aging: The Role of Multisensory Integration and Age-Related Sensory Loss

Brooks

Chan

Anderson

et al. 2018

Front. Hum. Neurosci.

View full text Add to dashboard Cite

show abstract

“…Forty years ago, in 1976, Harry McGurk and John McDonald discovered that by dubbing an auditory syllable (e.g., /ba/, hereafter the auditory component of a stimulus will be specified between slashes) with a different visual syllable (e.g., [ga], hereafter the visual component of a stimulus will be specified between brackets), the resulting auditory percept could be dramatically altered into a completely different syllable (e.g., ‘da’; see Massaro & Stork, ; for a description of how this discovery was made, and Yonovitz et al ., ; for a similar, independently achieved contemporary finding). This effect pushed the boundaries of audiovisual (AV) speech perception and multisensory integration by demonstrating that the influence of visual information on auditory speech perception goes beyond being a complement to the acoustic signal when it is degraded (e.g., Sumby & Pollack, ; Ross et al ., ; Jaekl et al ., ). Since then, the McGurk effect has been used in hundreds of studies to address the behavioral expression and physiological expression of multisensory integration in general and for AV speech integration in particular (Tiippana et al ., ; Alsius et al ., , ; Skipper et al ., ; van Wassenhove et al ., ; Bernstein et al ., ; Andersen et al ., ; Munhall et al ., ; Nahorna et al ., , ; Festa et al ., ).…”

Section: Introductionmentioning

confidence: 97%

Theta oscillations reflect conflict processing in the perception of the McGurk illusion

Fernández

Torralba

Soto‐Faraco

2018

Eur J of Neuroscience

Self Cite

View full text Add to dashboard Cite

The McGurk illusion is one of the most famous illustrations of cross-modal integration in human perception. It has been often used as a proxy of audiovisual (AV) integration and to infer the properties of the integration process in natural (congruent) AV conditions. Nonetheless, a blatant difference between McGurk stimuli and natural, congruent, AV speech is the conflict between the auditory and the visual information in the former. Here, we hypothesized that McGurk stimuli (and any AV incongruency) engage brain responses similar to those found in more general cases of perceptual conflict (e.g., Stroop), and propose that the McGurk illusion arises as a result of the resolution of such conflict. We used electroencephalography to measure variations in the power of theta, a well-known marker of the brain response to conflict. The results showed that perception of AV McGurk stimuli, just like AV incongruence in general, induces an increase in activity in the theta band. This response was similar to that evoked by Stroop stimuli, as measured in the same participants. This finding suggests that the McGurk illusion is mediated by general-purpose conflict mechanisms, and calls for caution in generalizing findings obtained using the McGurk illusion, to the general case of multisensory integration.

show abstract

“…Thus, measures of task performance under multisensory conditions show that multiple species can take advantage of the often complementary or redundant sensory information available to them in their environment (Bahrick & Lickliter, 2000; Foxe & Simpson, 2002; Gibson, 1969; Hammond-Kenny, Bajo, King, & Nodal, 2016; Stein, London, Wilkinson, & Price, 1996), allowing them to evolve and adapt to novel ecological niches (Karageorgi et al, 2017). In the case of humans, watching lip and facial movements, hand gestures, head nods, facial configurational (Jaekl, Pesquita, Alsius, Munhall, & Soto-Faraco, 2015) and even feeling the breath of a speaker on your skin (Gick & Derrick, 2009) can all provide additional information to an observer trying to understand what a speaker is saying to them (Ma, Zhou, Ross, Foxe, & Parra, 2009; Ross et al, 2011; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Sumby & Pollack, 1954). Even for more basic non-speech stimulus configurations, hearing a sound produced by a visual object is likely to enhance its detectability (Fiebelkorn et al, 2011; Molholm et al, 2002; Van der Burg, Olivers, Bronkhorst, & Theeuwes, 2008).…”

Section: Introductionmentioning

confidence: 99%

Ripe for solution: Delayed development of multisensory processing in autism and its remediation

Beker¹,

Foxe²,

Molholm³

2018

Neuroscience & Biobehavioral Reviews

View full text Add to dashboard Cite

Difficulty integrating inputs from different sensory sources is commonly reported in individuals with Autism Spectrum Disorder (ASD). Accumulating evidence consistently points to altered patterns of behavioral reactions and neural activity when individuals with ASD observe or act upon information arriving through multiple sensory systems. For example, impairments in the integration of seen and heard speech appear to be particularly acute, with obvious implications for interpersonal communication. Here, we explore the literature on multisensory processing in autism with a focus on developmental trajectories. While much remains to be understood, some consistent observations emerge. Broadly, sensory integration deficits are found in children with an ASD whereas these appear to be much ameliorated, or even fully recovered, in older teenagers and adults on the spectrum. This protracted delay in the development of multisensory processing raises the possibility of applying early intervention strategies focused on multisensory integration, to accelerate resolution of these functions. We also consider how dysfunctional cross-sensory oscillatory neural communication may be one key pathway to impaired multisensory processing in ASD.

show abstract

The contribution of dynamic visual cues to audiovisual speech perception

Cited by 14 publications

References 74 publications

Audiovisual Temporal Perception in Aging: The Role of Multisensory Integration and Age-Related Sensory Loss

Audiovisual Temporal Perception in Aging: The Role of Multisensory Integration and Age-Related Sensory Loss

Theta oscillations reflect conflict processing in the perception of the McGurk illusion

Ripe for solution: Delayed development of multisensory processing in autism and its remediation

Contact Info

Product

Resources

About