Computer Vision-Based Assessment of Autistic Children: Analyzing Interactions, Emotions, Human Pose, and Life Skills

Abstract

In this paper, the proposed work tests the computer vision application to perform the skill and emotion assessment of children with Autism Spectrum Disorder (ASD) by extracting various bio-behaviors, human activities, child-therapist interactions, and joint pose estimations from the video-recorded interactive singleor two-person play-based intervention sessions. A comprehensive data set of 300 videos are amassed from ASD children engaged in social interaction and developed three novel deep learning-based computer vision models which are explained as follows: 1) activity comprehension to analyze child-play partner interactions (Activity Comprehension model); 2) an automatic joint attention recognition framework using pose, and 3) emotion and facial expression recognition. We tested models on children's real-world unseen 68 videos captured from the clinic and public datasets. The activity comprehension model has an overall accuracy of 72.32%, the joint attention models have an accuracy of 97% for following eye gaze and 93.4% for hand pointing and the facial expression recognition model has an overall accuracy of 95.1%. The proposed models could extract activities and behaviors of interest from free-play and intervention session videos, empowering clinicians with data useful in diagnosis, assessment, treatment formulation, and monitoring of ASD children with limited supervision.

Systematic Review of Emotion Detection with Computer Vision and Deep Learning

“…Prakash et al [ 63 ] propose an approach for emotion detection in children with autism using videos. Their model combines a pre-trained ViT for gestures detection and a Resnet-34 model for facial macro-expression recognition.…”

A systematic review of computer vision technology applied for individuals with disabilities

“…where previous studies have analyzed the eye movements and behaviors of individuals with ASD to understand disability-related characteristics [4], and used related data for early screening and diagnosis [5].…”

Autism spectrum disorder (ASD) classification with three types of correlations based on ABIDE Ⅰ data

“…Resting-state functional magnetic resonance imaging (rs-fMRI), as a type of neuroimaging, has been becoming a popular approach to measuring and mapping brain activity since its system, which achieves higher AUC and specificity at 90% sensitivity with a sample size of 375. The study [28] employed three deep learning models on 300 videos of ASD children in social interactions. The activity comprehension model achieved 72.32% accuracy, joint attention recognition reached up to 97%, and facial expression recognition achieved 95.1% accuracy.…”