Crowdsourcing for Reference Correspondence Generation in Endoscopic Images

Crowdsourcing for error detection in cortical surface delineations

“…It has successfully been used in several medical imaging applications, and a review over medical applications using crowdsourcing is given in [19]. Examples of medical imaging tasks solved by crowdsourcing include disease detection based on optical microscopy images [17], shape-based classification of polyps in computed tomography data [18], medical image classification [3,11], reference data [8,15,16] and training data generation [14] as well as the assessments of surgical skills [4]. But it has-to our knowledge-not yet been applied in the context of neuroimaging data annotation.…”

Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences

“…Just like the KWs, the experts did not receive any training to complete the task. For a fair comparison, we ordered the images of Data Set I according to the median annotation performance obtained in an initial experiment [13] and then picked 10 images including the first and the last one (i.e., about every 11th image). This set of 100 features, each associated with 10 crowd annotations and five expert annotations, will be referred to as Data Set II.…”

“…For the data set applied in [13], the mean time required for obtaining 100 HITs (one per image) from MTurk was 77 ± 16 min, averaged over 10 requests (i.e., uploads of HITs). Hence, 10,000 annotations could be generated in about 12 h. The time from data upload until completion of 10,000 annotation tasks (10,000 for Data Set I and 2 × 10, 000 for Data Set II) in November 2014 ranged from 8 h 39 min to 13 h 10 min.…”

“…Exemplary tasks related to the last category include disease detection based on microscopic image analysis [8], shape-based classification of polyps in computed tomography data [9], fiber segmentation, 2 medical image classification [10], and surgical skill assessment [11] The goal of our work was to investigate whether crowdsourcing can be (part of) the solution to the large-scale data annotation problem in computer-assisted interventions. Two pilot studies in the context of instrument segmentation [12] and correspondence search [13], both presented at MIC-CAI 2014, showed that the quality of crowd annotations can compete with that of experts-but the crowd is orders of magnitude faster and also less expensive. This MICCAI special issue article is an extension of this work.…”

“…Herein, we propose a novel crowdsourcing-based approach to generating high-quality reference correspondences in pairs of endoscopic images without any expert interference, which is based on a novel method for outlier removal. We further present comprehensive validation studies involving a total of 30,000 new (crowdsourced) image annotations compared to the MICCAI contribution [13].…”