Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detection

Abstract

Standard medical diagnosis of mental health conditions requires licensed experts who are increasingly outnumbered by those at risk, limiting reach. We test the hypothesis that a trustworthy crowd of non-experts can efficiently annotate behavioral features needed for accurate machine learning detection of the common childhood developmental disorder Autism Spectrum Disorder (ASD) for children under 8 years old. We implement a novel process for identifying and certifying a trustworthy distributed workforce for video feature extraction, selecting a workforce of 102 workers from a pool of 1,107. Two previously validated ASD logistic regression classifiers, evaluated against parent-reported diagnoses, were used to assess the accuracy of the trusted crowd’s ratings of unstructured home videos. A representative balanced sample (N = 50 videos) of videos were evaluated with and without face box and pitch shift privacy alterations, with AUROC and AUPRC scores > 0.98. With both privacy-preserving modifications, sensitivity is preserved (96.0%) while maintaining specificity (80.0%) and accuracy (88.0%) at levels comparable to prior classification methods without alterations. We find that machine learning classification from features extracted by a certified nonexpert crowd achieves high performance for ASD detection from natural home videos of the child at risk and maintains high sensitivity when privacy-preserving mechanisms are applied. These results suggest that privacy-safeguarded crowdsourced analysis of short home videos can help enable rapid and mobile machine-learning detection of developmental delays in children.

Identification of social engagement indicators associated with autism spectrum disorder using a game-based mobile application

“…In addition, these techniques may compromise the privacy of participants by providing annotators with access to video footage, although some methods have been developed to address privacy concerns with crowdsourced annotations. 22,23…”

Crowd Annotations Can Approximate Clinical Autism Impressions from Short Home Videos with Privacy Protections

“…1,000 crowd workers completed the task and passed basic quality control checks for answer acceptance. Quality control measures included time spent on the annotation task and deviations in answers between videos [52]. We did not include workers who did not pass these basic quality control checks in the list of 1,000 evaluated workers.…”

“…This is important in general for healthcare applications, but it is especially critical when the patients in question are young children with a developmental delay and observed by a stranger in the privacy of their home. Our prior work has shown that applying light privacy-preserving modifications to videos, such as pitch shifting and covering the child's face with a virtual box, results in minimal degradation of the quality of crowdsourced annotations used for remote detection of autism-related behaviors [52]. However, such lightweight privacy protections may be insufficient to some patients, such as those patients who do not want the interior of their home exposed to strangers on the Internet.…”

“…Privacy rises to the forefront of patient concern when humans are incorporated into the diagnostic pipeline [56]. Many parents are uncomfortable with sharing raw videos of their children with strangers online even if the purpose of sharing is to help the parent's receive affordable and accessible diagnostic services [52]. However, human-in-the-loop techniques are pertinent given the current limitations of A.I., and privacy concerns must be directly addressed in any scalable solution where the humans in the loop are strangers (i.e., crowdsourced workers).…”

Identification of Social Engagement Indicators Associated With Autism Spectrum Disorder Using a Game-Based Mobile App: Comparative Study of Gaze Fixation and Visual Scanning Methods

“…Although these methods enable the creation of human-vetted, accurate data sets, such approaches require large numbers of trained annotators when implemented on a large scale, which is expensive and time-consuming. In addition, these techniques may compromise the privacy of participants by providing annotators with access to video footage, although some methods have been developed to address privacy concerns with crowdsourced annotations [ 23 , 24 ].…”