Selection of trustworthy crowd workers for telemedical diagnosis of pediatric autism spectrum disorder

Abstract

Crowd-powered telemedicine has the potential to revolutionize healthcare, especially during times that require remote access to care. However, sharing private health data with strangers from around the world is not compatible with data privacy standards, requiring a stringent filtration process to recruit reliable and trustworthy workers who can go through the proper training and security steps. The key challenge, then, is to identify capable, trustworthy, and reliable workers through high-fidelity evaluation tasks without exposing any sensitive patient data during the evaluation process. We contribute a set of experimentally validated metrics for assessing the trustworthiness and reliability of crowd workers tasked with providing behavioral feature tags to unstructured videos of children with autism and matched neurotypical controls. The workers are blinded to diagnosis and blinded to the goal of using the features to diagnose autism. These behavioral labels are fed as input to a previously validated binary logistic regression classifier for detecting autism cases using categorical feature vectors. While the metrics do not incorporate any ground truth labels of child diagnosis, linear regression using the 3 correlative metrics as input can predict the mean probability of the correct class of each worker with a mean average error of 7.51% for performance on the same set of videos and 10.93% for performance on a distinct balanced video set with different children. These results indicate that crowd workers can be recruited for performance based largely on behavioral metrics on a crowdsourced task, enabling an affordable way to filter crowd workforces into a trustworthy and reliable diagnostic workforce.

Crowdsourced feature tagging for scalable and privacy-preserved autism diagnosis

“…We are able to derive accurate diagnoses through feeding the crowd workers’ responses into a machine learning classifier. However, the recruitment of the crowd workforce was a crucial part of the process, as prior studies have shown that most crowd workers do not perform particularly well at labeling behavioral features from unstructured videos (54-56).…”

Identification of social engagement indicators associated with autism spectrum disorder using a game-based mobile application

“…In addition, these techniques may compromise the privacy of participants by providing annotators with access to video footage, although some methods have been developed to address privacy concerns with crowdsourced annotations. 22,23…”

Leveraging video data from a digital smartphone autism therapy to train an emotion detection classifier

“…can serve as a data acquisition tool and aggregate emotive videos for autism research that can be used to train a more effective automatic emotion recognition platform. The use of data collected from mobile devices, such as the built-in camera, allow for continuous phenotyping and repeat diagnoses in home settings [24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][39]. This motivates the development of a new emotion classifier designed specifically for pediatric populations, trained with images crowdsourced from Guess What?.…”