Feasibility of Automated Training for Facial Emotion Expression and Recognition in Autism

White, Susan W.; Abbott, Lynn; Wieckowski, Andrea Trubanova; Capriola-Hall, Nicole N.; Aly, Sherin; Youssef, Amira E.

doi:10.1016/j.beth.2017.12.010

Cited by 36 publications

(19 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, an AV training paradigm for speech recognition has shown improvement in speech intelligibility among hearing aid and CI users (Sato et al 2020). Complex skills such as face emotion recognition may also be enhanced in ASD after training using a 3D feedback game (White et al 2018). The aim of each of these novel interventions is to provide the user with redundant auditory and visual information in order to augment higherorder processes such as emotion recognition.…”

Section: Visual Facilitation Of Auditory Learning In Speech Perception and Social Communicationmentioning

confidence: 99%

Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

Opoku-Baah

Schoenhaut

Vassall

et al. 2021

JARO

View full text Add to dashboard Cite

In a naturalistic environment, auditory cues are often accompanied by information from other senses, which can be redundant with or complementary to the auditory information. Although the multisensory interactions derived from this combination of information and that shape auditory function are seen across all sensory modalities, our greatest body of knowledge to date centers on how vision influences audition. In this review, we attempt to capture the state of our understanding at this point in time regarding this topic. Following a general introduction, the review is divided into 5 sections. In the first section, we review the psychophysical evidence in humans regarding vision’s influence in audition, making the distinction between vision’s ability to enhance versus alter auditory performance and perception. Three examples are then described that serve to highlight vision’s ability to modulate auditory processes: spatial ventriloquism, cross-modal dynamic capture, and the McGurk effect. The final part of this section discusses models that have been built based on available psychophysical data and that seek to provide greater mechanistic insights into how vision can impact audition. The second section reviews the extant neuroimaging and far-field imaging work on this topic, with a strong emphasis on the roles of feedforward and feedback processes, on imaging insights into the causal nature of audiovisual interactions, and on the limitations of current imaging-based approaches. These limitations point to a greater need for machine-learning-based decoding approaches toward understanding how auditory representations are shaped by vision. The third section reviews the wealth of neuroanatomical and neurophysiological data from animal models that highlights audiovisual interactions at the neuronal and circuit level in both subcortical and cortical structures. It also speaks to the functional significance of audiovisual interactions for two critically important facets of auditory perception—scene analysis and communication. The fourth section presents current evidence for alterations in audiovisual processes in three clinical conditions: autism, schizophrenia, and sensorineural hearing loss. These changes in audiovisual interactions are postulated to have cascading effects on higher-order domains of dysfunction in these conditions. The final section highlights ongoing work seeking to leverage our knowledge of audiovisual interactions to develop better remediation approaches to these sensory-based disorders, founded in concepts of perceptual plasticity in which vision has been shown to have the capacity to facilitate auditory learning.

show abstract

Section: Visual Facilitation Of Auditory Learning In Speech Perception and Social Communicationmentioning

confidence: 99%

Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

Opoku-Baah

Schoenhaut

Vassall

et al. 2021

JARO

View full text Add to dashboard Cite

show abstract

“…Harrold et al 54,55 developed a mobile application that allows children to learn emotions with instant feedback on performance through computer vision. White et al 56 presented results which showed that children with ASD found their system to be acceptable and enjoyable. Similar to this approach, Garcia-Garcia et al 57 presented a system that incorporates emotion recognition and tangible user interfaces to teach children with ASD to identify and express emotions.…”

Section: Facial Expression/emotionmentioning

confidence: 99%

Computer vision in autism spectrum disorder research: a systematic review of published studies from 2009 to 2019

et al. 2020

View full text Add to dashboard Cite

The current state of computer vision methods applied to autism spectrum disorder (ASD) research has not been well established. Increasing evidence suggests that computer vision techniques have a strong impact on autism research. The primary objective of this systematic review is to examine how computer vision analysis has been useful in ASD diagnosis, therapy and autism research in general. A systematic review of publications indexed on PubMed, IEEE Xplore and ACM Digital Library was conducted from 2009 to 2019. Search terms included [‘autis*’ AND (‘computer vision’ OR ‘behavio* imaging’ OR ‘behavio* analysis’ OR ‘affective computing’)]. Results are reported according to PRISMA statement. A total of 94 studies are included in the analysis. Eligible papers are categorised based on the potential biological/behavioural markers quantified in each study. Then, different computer vision approaches that were employed in the included papers are described. Different publicly available datasets are also reviewed in order to rapidly familiarise researchers with datasets applicable to their field and to accelerate both new behavioural and technological work on autism research. Finally, future research directions are outlined. The findings in this review suggest that computer vision analysis is useful for the quantification of behavioural/biological markers which can further lead to a more objective analysis in autism research.

show abstract

“…To the best of our knowledge, there are no published works exploiting computer vision to quantitatively assess the ability of children with ASD to produce facial expressions. Invasive methods (based on eye tracker and electroencephalography) [ 26 ] or questionnaire measures [ 27 ] have been successfully applied to this aim instead, providing scientific proofs that the production of expressions can be strengthened through practice and training.…”

Section: Introductionmentioning

confidence: 99%

Computational Assessment of Facial Expression Production in ASD Children

Leo

Carcagnì

Distante

et al. 2018

Sensors

View full text Add to dashboard Cite

In this paper, a computational approach is proposed and put into practice to assess the capability of children having had diagnosed Autism Spectrum Disorders (ASD) to produce facial expressions. The proposed approach is based on computer vision components working on sequence of images acquired by an off-the-shelf camera in unconstrained conditions. Action unit intensities are estimated by analyzing local appearance and then both temporal and geometrical relationships, learned by Convolutional Neural Networks, are exploited to regularize gathered estimates. To cope with stereotyped movements and to highlight even subtle voluntary movements of facial muscles, a personalized and contextual statistical modeling of non-emotional face is formulated and used as a reference. Experimental results demonstrate how the proposed pipeline can improve the analysis of facial expressions produced by ASD children. A comparison of system’s outputs with the evaluations performed by psychologists, on the same group of ASD children, makes evident how the performed quantitative analysis of children’s abilities helps to go beyond the traditional qualitative ASD assessment/diagnosis protocols, whose outcomes are affected by human limitations in observing and understanding multi-cues behaviors such as facial expressions.

show abstract

Feasibility of Automated Training for Facial Emotion Expression and Recognition in Autism

Cited by 36 publications

References 30 publications

Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

Computer vision in autism spectrum disorder research: a systematic review of published studies from 2009 to 2019

Computational Assessment of Facial Expression Production in ASD Children

Contact Info

Product

Resources

About