A Survey of Crowdsourcing in Medical Image Analysis

Abstract

Rapid advances in image processing capabilities have been seen across many domains, fostered by the application of machine learning algorithms to "big-data". However, within the realm of medical image analysis, advances have been curtailed, in part, due to the limited availability of large-scale, well-annotated datasets. One of the main reasons for this is the high cost often associated with producing large amounts of high-quality meta-data. Recently, there has been growing interest in the application of crowdsourcing for this purpose; a technique that has proven effective for creating large-scale datasets across a range of disciplines, from computer vision to astrophysics. Despite the growing popularity of this approach, there has not yet been a comprehensive literature review to provide guidance to researchers considering using crowdsourcing methodologies in their own medical imaging analysis. In this survey, we review studies applying crowdsourcing to the analysis of medical images, published prior to July 2018. We identify common approaches, challenges and considerations, providing guidance of utility to researchers adopting this approach. Finally, we discuss future opportunities for development within this emerging domain.

E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation

“…Our results provide justification toward using large-scale nonexpert contours for gold-standard segmentation data in the absence of multiple expert “ground truth” data availability, and glean insight into the behavior of consensus contours across a large number of observer inputs. Although crowdsourcing is common in medical image analysis, 31 there have been few studies evaluating the use of crowdsourcing for contour quality. To our knowledge, this is the largest study characterizing segmentation performance across multiple physician observers, and the first study to investigate crowdsourced contour performance in the context of radiation oncology workflows.…”

Effective and efficient active learning for deep learning-based tissue image analysis

“…Several projects have developed techniques to generate synthetic training datasets, consisting of pairs of synthetic tissue images and masks, for image segmentation ( Hossain and Sakib 2020 ; Gong et al 2021 ). Crowdsourcing approaches also have been implemented to curate large training datasets ( Li and Deng 2018 ; Ørting et al 2019) .…”

Online citizen science with the Zooniverse for analysis of biological volumetric data

“…Multiple projects to date have demonstrated that citizen scientists can be effectively engaged in a wide variety of biological image analysis tasks, such as the tracing of connected neurons through electron microscopy (EM) data in Eyewire (Kim et al, 2014), clicking on protein particles in cryo-EM data in Microscopy Masters (Bruggemann et al, 2018), and scoring tumor markers in pathology samples in Cell slider (Dos Reis et al, 2015), amongst others (Smittenaar et al, 2018;Ørting et al, 2019;Benhajali et al, 2020;Fisch et al, 2021;Fowler et al, 2022). The efficacy of applying online citizen science for the analysis of biological image data, both as an avenue for producing novel research and as a powerful tool for public engagement, has led to a rise in interest from the biological volumetric imaging research community in applying this methodology.…”