Training and Profiling a Pediatric Facial Expression Classifier for Children on Mobile Devices: Machine Learning Study

Abstract

Background Implementing automated facial expression recognition on mobile devices could provide an accessible diagnostic and therapeutic tool for those who struggle to recognize facial expressions, including children with developmental behavioral conditions such as autism. Despite recent advances in facial expression classifiers for children, existing models are too computationally expensive for smartphone use. Objective We explored several state-of-the-art facial expression classifiers designed for mobile devices, used posttraining optimization techniques for both classification performance and efficiency on a Motorola Moto G6 phone, evaluated the importance of training our classifiers on children versus adults, and evaluated the models’ performance against different ethnic groups. Methods We collected images from 12 public data sets and used video frames crowdsourced from the GuessWhat app to train our classifiers. All images were annotated for 7 expressions: neutral, fear, happiness, sadness, surprise, anger, and disgust. We tested 3 copies for each of 5 different convolutional neural network architectures: MobileNetV3-Small 1.0x, MobileNetV2 1.0x, EfficientNetB0, MobileNetV3-Large 1.0x, and NASNetMobile. We trained the first copy on images of children, second copy on images of adults, and third copy on all data sets. We evaluated each model against the entire Child Affective Facial Expression (CAFE) set and by ethnicity. We performed weight pruning, weight clustering, and quantize-aware training when possible and profiled each model’s performance on the Moto G6. Results Our best model, a MobileNetV3-Large network pretrained on ImageNet, achieved 65.78% accuracy and 65.31% F1-score on the CAFE and a 90-millisecond inference latency on a Moto G6 phone when trained on all data. This accuracy is only 1.12% lower than the current state of the art for CAFE, a model with 13.91x more parameters that was unable to run on the Moto G6 due to its size, even when fully optimized. When trained solely on children, this model achieved 60.57% accuracy and 60.29% F1-score. When trained only on adults, the model received 53.36% accuracy and 53.10% F1-score. Although the MobileNetV3-Large trained on all data sets achieved nearly a 60% F1-score across all ethnicities, the data sets for South Asian and African American children achieved lower accuracy (as much as 11.56%) and F1-score (as much as 11.25%) than other groups. Conclusions With specialized design and optimization techniques, facial expression classifiers can become lightweight enough to run on mobile devices and achieve state-of-the-art performance. There is potentially a “data shift” phenomenon between facial expressions of children compared with adults; our classifiers performed much better when trained on children. Certain underrepresented ethnic groups (e.g., South Asian and African American) also perform significantly worse than groups such as European Caucasian despite similar data quality. Our models can be integrated into mobile health therapies to help diagnose autism spectrum disorder and provide targeted therapeutic treatment to children.

Artificial Intelligence and Pediatrics: Synthetic Knowledge Synthesis

“…Diagnosis of autism spectrum disorder with artificial intelligence: Convolutional neural networks were used to develop an automated facial expression recognition therapeutic tool on mobile devices for children with autism [63]. A method based on a radial basis function neural network was used to support the design and evaluation of educational toys for children with autism [64].…”

Personalized Deep Learning for Substance Use in Hawaii: Protocol for a Passive Sensing and Ecological Momentary Assessment Study

“…This setting of large, unlabeled data sets with sparse supervision appears frequently in the field of digital health care. Notable examples include passive mobile sensing studies for mental health and well-being [11][12][13][14][15][16][17][18][19][20], digital therapeutics for children with autism spectrum disorder that record videos of the child [21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37], and passive brain sensors for brain-computer interfaces [38][39][40][41][42][43][44][45][46]. As such, this study protocol can be considered as one of the first tests of a broader emerging paradigm in precision health.…”

#ActuallyAutistic Twitter dataset for precision diagnosis of Autism Spectrum Disorder (ASD)

“…Research in mental health such as identifying depression and mood changes [8][9][10][11][12][13][14] , and real-time mapping of natural disasters 15,16 or infectious disease spread and its effect on emotional health [17][18][19][20][21][22][23][24] has greatly benefited from digital phenotyping. ASD has been the subject of multiple clinical trials, reviews, and epidemiological studies conducted using behavioral features such as eye gaze 25 , prosody 26 , asynchronous body movement 27 , facial expressions 28,29 , mobile phone data [30][31][32][33] or even electroencephalogram (EEG) 34 . However, only a handful of studies have used social analytical tools [35][36][37][38] , especially using Twitter 39,40,41 for investigating ASD.…”