Identification of Social Engagement Indicators Associated With Autism Spectrum Disorder Using a Game-Based Mobile App: Comparative Study of Gaze Fixation and Visual Scanning Methods

Abstract

Background Autism spectrum disorder (ASD) is a widespread neurodevelopmental condition with a range of potential causes and symptoms. Standard diagnostic mechanisms for ASD, which involve lengthy parent questionnaires and clinical observation, often result in long waiting times for results. Recent advances in computer vision and mobile technology hold potential for speeding up the diagnostic process by enabling computational analysis of behavioral and social impairments from home videos. Such techniques can improve objectivity and contribute quantitatively to the diagnostic process. Objective In this work, we evaluate whether home videos collected from a game-based mobile app can be used to provide diagnostic insights into ASD. To the best of our knowledge, this is the first study attempting to identify potential social indicators of ASD from mobile phone videos without the use of eye-tracking hardware, manual annotations, and structured scenarios or clinical environments. Methods Here, we used a mobile health app to collect over 11 hours of video footage depicting 95 children engaged in gameplay in a natural home environment. We used automated data set annotations to analyze two social indicators that have previously been shown to differ between children with ASD and their neurotypical (NT) peers: (1) gaze fixation patterns, which represent regions of an individual’s visual focus and (2) visual scanning methods, which refer to the ways in which individuals scan their surrounding environment. We compared the gaze fixation and visual scanning methods used by children during a 90-second gameplay video to identify statistically significant differences between the 2 cohorts; we then trained a long short-term memory (LSTM) neural network to determine if gaze indicators could be predictive of ASD. Results Our results show that gaze fixation patterns differ between the 2 cohorts; specifically, we could identify 1 statistically significant region of fixation (P<.001). In addition, we also demonstrate that there are unique visual scanning patterns that exist for individuals with ASD when compared to NT children (P<.001). A deep learning model trained on coarse gaze fixation annotations demonstrates mild predictive power in identifying ASD. Conclusions Ultimately, our study demonstrates that heterogeneous video data sets collected from mobile devices hold potential for quantifying visual patterns and providing insights into ASD. We show the importance of automated labeling techniques in generating large-scale data sets while simultaneously preserving the privacy of participants, and we demonstrate that specific social engagement indicators associated with ASD can be identified and characterized using such data.

Parkinson Disease Recognition Using a Gamified Website: Machine Learning Development and Usability Study

“…Another potentially fruitful avenue of expanding PD screening tools would be to include additional data modalities such as computer vision. Computer vision analysis has been successfully used for a variety of health screening and diagnostic tasks, including abnormal hand movements and movement of other body parts for conditions such as autism [ 61 - 67 ]. Using such techniques for PD screening can expand the performance of the tools through a more comprehensive and multimodal analysis.…”

A Review of and Roadmap for Data Science and Machine Learning for the Neuropsychiatric Phenotype of Autism

“…They found that there were statistically significant differences between children with autism and neurotypical controls in the degree centrality for areas pertaining to the mouth and right eye. Breaking from the more standard practice of using highly structured interactions and in-lab settings for data collection, Varma et al ( 44 ) measured eye gaze differences between cohorts using network analysis from crowdsourced data collected during use of a mobile autism therapeutic, finding a statistically significant difference between the groups for a single area of interest. Alvari et al ( 45 ) analyzed eye contact during unconstrained therapist–child interactions by applying unsupervised clustering on data from 62 children with autism, identifying three distinct subgroups defined by eye contact dynamics.…”

#ActuallyAutistic Twitter dataset for precision diagnosis of Autism Spectrum Disorder (ASD)

“…Research in mental health such as identifying depression and mood changes [8][9][10][11][12][13][14] , and real-time mapping of natural disasters 15,16 or infectious disease spread and its effect on emotional health [17][18][19][20][21][22][23][24] has greatly benefited from digital phenotyping. ASD has been the subject of multiple clinical trials, reviews, and epidemiological studies conducted using behavioral features such as eye gaze 25 , prosody 26 , asynchronous body movement 27 , facial expressions 28,29 , mobile phone data [30][31][32][33] or even electroencephalogram (EEG) 34 . However, only a handful of studies have used social analytical tools [35][36][37][38] , especially using Twitter 39,40,41 for investigating ASD.…”