Leveraging video data from a digital smartphone autism therapy to train an emotion detection classifier

Abstract

Autism spectrum disorder (ASD) is a neurodevelopmental disorder affecting one in 40 children in the United States and is associated with impaired social interactions, restricted interests, and repetitive behaviors. Previous studies have demonstrated the promise of applying mobile systems with real-time emotion recognition to autism therapy, but existing platforms have shown limited performance on videos of children with ASD. We propose the development of a new emotion classifier designed specifically for pediatric populations, trained with images crowdsourced from an educational mobile charades-style game: Guess What?. We crowdsourced the acquisition of videos of children portraying emotions during remote game sessions of Guess What? that yielded 6,344 frames from fifteen subjects. Two raters manually labeled the frames with four of the Ekman universal emotions (happy, scared, angry, sad), a neutral class, and n/a for frames with an indeterminable label. The data were pre-processed, and a model was trained with a transfer-learning and neural-architecture-search approach using the Google Cloud AutoML Vision API. The resulting classifier was evaluated against existing approaches (Azure Face API from Microsoft and Rekognition from Amazon Web Services) using the standard metrics of F1 score. The resulting classifier demonstrated superior performance across all evaluated emotions, supporting our hypothesis that a model trained with a pediatric dataset would outperform existing emotion-recognition approaches for the population of interest. These results suggest a new strategy to develop precision therapy for autism at home by integrating the model trained with a personalized dataset to the mobile game.

Computer Vision Estimation of Stress and Anxiety Using a Gamified Mobile-based Ecological Momentary Assessment and Deep Learning: Research Protocol

“…Our plan is to develop the STAND app into a versatile tool that can be customized for researching and gathering computer vision data of the face to build personalized machine learning models which can support digital health interventions related to mental states and developmental disorders. This work can integrate with existing research in using automatic emotion recognition for a variety of contexts, including mental illness diagnosis, recognizing human social and physiological interactions, and developing sociable robotics and other human-computer interaction systems [57, [85][86][87][88][89]. For example, emotional expressions have a crucial role in recognizing certain types of developmental disorders.…”

Classifying Autism From Crowdsourced Semistructured Speech Recordings: Machine Learning Model Comparison Study

“…Previous work examined the use of crowdsourced annotations for autism, indicating that similar approaches could perhaps be applied through audio [31,[46][47][48][49][50][51]. Audio feature extraction combined with other autism classifiers could be used to create an explainable diagnostic system [52][53][54][55][56][57][58][59][60][61][62][63][64] fit for mobile devices [60]. Previous work investigated using such classifiers to detect autism or approach autism-related tasks like identifying emotion to improve socialization skills; combining computer vision-based quantification of relevant areas of interest, including hand stimming [58], upper limb movement [63], and eye contact [62,64], could possibly result in interpretable models.…”

A Review of and Roadmap for Data Science and Machine Learning for the Neuropsychiatric Phenotype of Autism

“…sessions with emotion game prompts were demonstrated to be enriched with emotional facial expressions (22). The resulting frames enabled the construction of models that outperformed general-purpose facial emotion recognition classifiers (23)(24)(25)(26). Beyond its value for digital phenotyping, Guess What?…”