Crowdsourced feature tagging for scalable and privacy-preserved autism diagnosis

Washington, Peter; Tariq, Qandeel; Leblanc, Émilie; Chrisman, Brianna; Dunlap, Kaitlyn; Kline, Aaron; Kalantarian, Haik; Penev, Yordan; Paskov, Kelley; Voss, Catalin; Varma, Maya; Husic, Arman; Kent, Jack; Haber, Nick; Winograd, Terry; Wall, Dennis P.

doi:10.1101/2020.12.15.20248283

Cited by 6 publications

(2 citation statements)

References 54 publications

(60 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Clinicians spend several hours measuring dozens of behavioral features when making a diagnosis [9], further accounting for the long wait times that make it difficult to get an appointment. However, prior research has shown that machine learning models can achieve similar diagnostic capabilities for children with ASD [10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26], providing rapid inference using fewer than 10 behavioral features that can be easily collected through mediums such as short video clips [16,[26][27][28][29]. Models that analyze a single ASD-related symptom such as speech patterns [30], hand stimming [31], and head banging [32] have provided promising results for diagnosis of ASD when tested on highly heterogeneous data from real children.…”

Section: Introductionmentioning

confidence: 99%

Training and Profiling a Pediatric Facial Expression Classifier for Children on Mobile Devices: Machine Learning Study

Banerjee¹,

Mutlu²,

Kline³

et al. 2023

JMIR Form Res

Self Cite

View full text Add to dashboard Cite

Background Implementing automated facial expression recognition on mobile devices could provide an accessible diagnostic and therapeutic tool for those who struggle to recognize facial expressions, including children with developmental behavioral conditions such as autism. Despite recent advances in facial expression classifiers for children, existing models are too computationally expensive for smartphone use. Objective We explored several state-of-the-art facial expression classifiers designed for mobile devices, used posttraining optimization techniques for both classification performance and efficiency on a Motorola Moto G6 phone, evaluated the importance of training our classifiers on children versus adults, and evaluated the models’ performance against different ethnic groups. Methods We collected images from 12 public data sets and used video frames crowdsourced from the GuessWhat app to train our classifiers. All images were annotated for 7 expressions: neutral, fear, happiness, sadness, surprise, anger, and disgust. We tested 3 copies for each of 5 different convolutional neural network architectures: MobileNetV3-Small 1.0x, MobileNetV2 1.0x, EfficientNetB0, MobileNetV3-Large 1.0x, and NASNetMobile. We trained the first copy on images of children, second copy on images of adults, and third copy on all data sets. We evaluated each model against the entire Child Affective Facial Expression (CAFE) set and by ethnicity. We performed weight pruning, weight clustering, and quantize-aware training when possible and profiled each model’s performance on the Moto G6. Results Our best model, a MobileNetV3-Large network pretrained on ImageNet, achieved 65.78% accuracy and 65.31% F1-score on the CAFE and a 90-millisecond inference latency on a Moto G6 phone when trained on all data. This accuracy is only 1.12% lower than the current state of the art for CAFE, a model with 13.91x more parameters that was unable to run on the Moto G6 due to its size, even when fully optimized. When trained solely on children, this model achieved 60.57% accuracy and 60.29% F1-score. When trained only on adults, the model received 53.36% accuracy and 53.10% F1-score. Although the MobileNetV3-Large trained on all data sets achieved nearly a 60% F1-score across all ethnicities, the data sets for South Asian and African American children achieved lower accuracy (as much as 11.56%) and F1-score (as much as 11.25%) than other groups. Conclusions With specialized design and optimization techniques, facial expression classifiers can become lightweight enough to run on mobile devices and achieve state-of-the-art performance. There is potentially a “data shift” phenomenon between facial expressions of children compared with adults; our classifiers performed much better when trained on children. Certain underrepresented ethnic groups (e.g., South Asian and African American) also perform significantly worse than groups such as European Caucasian despite similar data quality. Our models can be integrated into mobile health therapies to help diagnose autism spectrum disorder and provide targeted therapeutic treatment to children.

show abstract

Section: Introductionmentioning

confidence: 99%

Training and Profiling a Pediatric Facial Expression Classifier for Children on Mobile Devices: Machine Learning Study

Banerjee¹,

Mutlu²,

Kline³

et al. 2023

JMIR Form Res

Self Cite

View full text Add to dashboard Cite

show abstract

“…Mobile diagnostic efforts for autism using machine learning have been explored in prior literature. Autism can be classified with high performance using 10 or fewer behavioral features [23][24][25][26][27][28]. While some untrained humans can reliably distinguish these behavioral features [25,[29][30][31][32][33][34][35][36], an eventual goal is to move away from human-in-the-loop solutions toward automated and privacy-preserving diagnostic solutions [37,38].…”

Section: Introductionmentioning

confidence: 99%

The Classification of Abnormal Hand Movement to Aid in Autism Detection: Machine Learning Study

Lakkapragada¹,

Kline²,

Mutlu³

et al. 2022

JMIR Biomed Eng

Self Cite

View full text Add to dashboard Cite

Background A formal autism diagnosis can be an inefficient and lengthy process. Families may wait several months or longer before receiving a diagnosis for their child despite evidence that earlier intervention leads to better treatment outcomes. Digital technologies that detect the presence of behaviors related to autism can scale access to pediatric diagnoses. A strong indicator of the presence of autism is self-stimulatory behaviors such as hand flapping. Objective This study aims to demonstrate the feasibility of deep learning technologies for the detection of hand flapping from unstructured home videos as a first step toward validation of whether statistical models coupled with digital technologies can be leveraged to aid in the automatic behavioral analysis of autism. To support the widespread sharing of such home videos, we explored privacy-preserving modifications to the input space via conversion of each video to hand landmark coordinates and measured the performance of corresponding time series classifiers. Methods We used the Self-Stimulatory Behavior Dataset (SSBD) that contains 75 videos of hand flapping, head banging, and spinning exhibited by children. From this data set, we extracted 100 hand flapping videos and 100 control videos, each between 2 to 5 seconds in duration. We evaluated five separate feature representations: four privacy-preserved subsets of hand landmarks detected by MediaPipe and one feature representation obtained from the output of the penultimate layer of a MobileNetV2 model fine-tuned on the SSBD. We fed these feature vectors into a long short-term memory network that predicted the presence of hand flapping in each video clip. Results The highest-performing model used MobileNetV2 to extract features and achieved a test F1 score of 84 (SD 3.7; precision 89.6, SD 4.3 and recall 80.4, SD 6) using 5-fold cross-validation for 100 random seeds on the SSBD data (500 total distinct folds). Of the models we trained on privacy-preserved data, the model trained with all hand landmarks reached an F1 score of 66.6 (SD 3.35). Another such model trained with a select 6 landmarks reached an F1 score of 68.3 (SD 3.6). A privacy-preserved model trained using a single landmark at the base of the hands and a model trained with the average of the locations of all the hand landmarks reached an F1 score of 64.9 (SD 6.5) and 64.2 (SD 6.8), respectively. Conclusions We created five lightweight neural networks that can detect hand flapping from unstructured videos. Training a long short-term memory network with convolutional feature vectors outperformed training with feature vectors of hand coordinates and used almost 900,000 fewer model parameters. This study provides the first step toward developing precise deep learning methods for activity detection of autism-related behaviors.

show abstract

Feature replacement methods enable reliable home video analysis for machine learning detection of autism

Leblanc

Washington

Varma

et al. 2020

Sci Rep

Self Cite

View full text Add to dashboard Cite

Autism Spectrum Disorder is a neuropsychiatric condition affecting 53 million children worldwide and for which early diagnosis is critical to the outcome of behavior therapies. Machine learning applied to features manually extracted from readily accessible videos (e.g., from smartphones) has the potential to scale this diagnostic process. However, nearly unavoidable variability in video quality can lead to missing features that degrade algorithm performance. To manage this uncertainty, we evaluated the impact of missing values and feature imputation methods on two previously published autism detection classifiers, trained on standard-of-care instrument scoresheets and tested on ratings of 140 children videos from YouTube. We compare the baseline method of listwise deletion to classic univariate and multivariate techniques. We also introduce a feature replacement method that, based on a score, selects a feature from an expanded dataset to fill-in the missing value. The replacement feature selected can be identical for all records (general) or automatically adjusted to the record considered (dynamic). Our results show that general and dynamic feature replacement methods achieve a higher performance than classic univariate and multivariate methods, supporting the hypothesis that algorithmic management can maintain the fidelity of video-based diagnostics in the face of missing values and variable video quality.

show abstract

Crowdsourced feature tagging for scalable and privacy-preserved autism diagnosis

Cited by 6 publications

References 54 publications

Training and Profiling a Pediatric Facial Expression Classifier for Children on Mobile Devices: Machine Learning Study

Training and Profiling a Pediatric Facial Expression Classifier for Children on Mobile Devices: Machine Learning Study

The Classification of Abnormal Hand Movement to Aid in Autism Detection: Machine Learning Study

Feature replacement methods enable reliable home video analysis for machine learning detection of autism

Contact Info

Product

Resources

About