The Accuracy and Reliability of Crowdsource Annotations of Digital Retinal Images

Abstract

PurposeCrowdsourcing is based on outsourcing computationally intensive tasks to numerous individuals in the online community who have no formal training. Our aim was to develop a novel online tool designed to facilitate large-scale annotation of digital retinal images, and to assess the accuracy of crowdsource grading using this tool, comparing it to expert classification.MethodsWe used 100 retinal fundus photograph images with predetermined disease criteria selected by two experts from a large cohort study. The Amazon Mechanical Turk Web platform was used to drive traffic to our site so anonymous workers could perform a classification and annotation task of the fundus photographs in our dataset after a short training exercise. Three groups were assessed: masters only, nonmasters only and nonmasters with compulsory training. We calculated the sensitivity, specificity, and area under the curve (AUC) of receiver operating characteristic (ROC) plots for all classifications compared to expert grading, and used the Dice coefficient and consensus threshold to assess annotation accuracy.ResultsIn total, we received 5389 annotations for 84 images (excluding 16 training images) in 2 weeks. A specificity and sensitivity of 71% (95% confidence interval [CI], 69%–74%) and 87% (95% CI, 86%–88%) was achieved for all classifications. The AUC in this study for all classifications combined was 0.93 (95% CI, 0.91–0.96). For image annotation, a maximal Dice coefficient (∼0.6) was achieved with a consensus threshold of 0.25.ConclusionsThis study supports the hypothesis that annotation of abnormalities in retinal images by ophthalmologically naive individuals is comparable to expert annotation. The highest AUC and agreement with expert annotation was achieved in the nonmasters with compulsory training group.Translational RelevanceThe use of crowdsourcing as a technique for retinal image analysis may be comparable to expert graders and has the potential to deliver timely, accurate, and cost-effective image analysis.

Crowdsourcing of Histological Image Labeling and Object Delineation by Medical Students

“…Other studies focus on the creation of training sets for convolutional neural networks for finding nuclei or mitoses in cancer [8], [9]. Crowdsourcing has also been applied in labeling of retinal images [10], text annotation in radiology reports [11], or for delineation of a single object per image [12].…”

Validation and tuning ofin situtranscriptomics image processing workflows with crowdsourced annotations

“…When tasks are less intrinsically interesting to volunteers, minimally-trained workers can complete tasks for small payments through crowdsourcing platforms such as Amazon's Mechanical Turk (MTurk), and the consensus annotations (across multiple workers or "turkers") can be highly comparable with expert annotations, and sufficiently reliable for use as training data for detection algorithms. (27)(28)(29) Therefore, we hypothesized that consensus from crowdsourced annotations can be used as a substitute for ground truth to tune and benchmark spot-calling algorithms. However, there are no published in situ transcriptomics pipelines that can incorporate ground truth from crowdsourced annotations.…”

Multi-Dimensional Machine Learning Approaches for Fruit Shape Recognition and Phenotyping in Strawberry

“…Categories may be either nominal [11, 18, 19], existing in name only, or ordinal, referring to a position in an ordered series or on a gradient [15, 16, 19]. Classification is often labor-intensive and prone to human bias, which can increase with task complexity and time requirement [20, 21]. Alternative scoring approaches have relied on morphometrics and machine learning to automate classification; for example, sorting fruit into shape categories in both tomato [11] and strawberry [18].…”