Crowdsourcing lung nodules detection and annotation

Abstract

We present crowdsourcing as an additional modality to aid radiologists in the diagnosis of lung cancer from clinical chest computed tomography (CT) scans. More specifically, a complete workflow is introduced which can help maximize the sensitivity of lung nodule detection by utilizing the collective intelligence of the crowd. We combine the concept of overlapping thin-slab maximum intensity projections (TS-MIPs) and cine viewing to render short videos that can be outsourced as an annotation task to the crowd. These videos are generated by linearly interpolating overlapping TS-MIPs of CT slices through the depth of each quadrant of a patient's lung. The resultant videos are outsourced to an online community of non-expert users who, after a brief tutorial, annotate suspected nodules in these video segments. Using our crowdsourcing workflow, we achieved a lung nodule detection sensitivity of over 90% for 20 patient CT datasets (containing 178 lung nodules with sizes between 1-30mm), and only 47 false positives from a total of 1021 annotations on nodules of all sizes (96% sensitivity for nodules>4mm). These results show that crowdsourcing can be a robust and scalable modality to aid radiologists in screening for lung cancer, directly or in combination with computer-aided detection (CAD) algorithms. For CAD algorithms, the presented workflow can provide highly accurate training data to overcome the high false-positive rate (per scan) problem. We also provide, for the first time, analysis on nodule size and position which can help improve CAD algorithms.

Development of a volumetric pancreas segmentation CT dataset for AI applications through trained technologists: a study during the COVID 19 containment phase

“…In the literature, experiences with crowdsourcing medical imaging tasks to untrained persons in the community-at-large have been described with variable success. Such tasks include annotations of airways, lung nodules, kidney and liver segmentations, and colon polyp classification on CT colonography images [18][19][20][21].…”

A Survey of Crowdsourcing in Medical Image Analysis

“…Classify (Albarqouni et al, 2016a), (Brady et al, 2014), (Brady et al, 2017), , (dos Reis et al, 2015), (Eickhoff, 2014), (Foncubierta Rodríguez and Müller, 2012), (Gur et al, 2017), (de Herrera et al, 2014), (Holst et al, 2015), (Huang and Hamarneh, 2017), (Keshavan et al, 2018), (Lawson et al, 2017), (Malpani et al, 2015), (Mavandadi et al, 2012, (Mitry et al, 2013), (Mitry et al, 2015), (Nguyen et al, 2012), (Park et al, 2016), (Park et al, 2017), (Smittenaar et al, 2018), (Sonabend et al, 2017), (Sullivan et al, 2018) Segment (Roethlingshoefer et al, 2017), (Boorboor et al, 2018), (Bruggemann et al, 2018), (Cabrera-Bean et al, 2017), (Chávez-Aragón et al, 2013), (Cheplygina et al, 2016), (Ganz et al, 2017), (Gurari et al, 2015b), (Heller et al, 2017), (Irshad et al, 2015), (Lee and Tufail, 2014), (Lee et al, 2016), (Lejeune et al, 2017), (Luengo-Oroz et al, 2012), (Maier-Hein et al, 2014a), (Maier-Hein et al, 2016), (O'Neil et al, 2017), (Park et al, 2018),…”

“…Click (Bruggemann et al, 2018), (Cabrera-Bean et al, 2017), (Della Mea et al, 2014), (Huang and Hamarneh, 2017), (Lawson et al, 2017), (Lejeune et al, 2017), (Luengo-Oroz et al, 2012), (Park et al, 2018), (Rajchl et al, 2016) Click + Compare (Maier-Hein et al, 2014b), (Maier-Hein et al, 2015), (Maier-Hein et al, 2016) Click + Draw (Irshad et al, 2015) Compare (Ørting et al, 2017) Draw (Roethlingshoefer et al, 2017), (Boorboor et al, 2018), (Chávez-Aragón et al, 2013), (Cheplygina et al, 2016), (Ganz et al, 2017), (Gurari et al, 2015b), (Heller et al, 2017), (Lee and Tufail, 2014), (Lee et al, 2016), (Maier-Hein et al, 2014a), (O'Neil et al, 2017), (Sameki et al, 2016), (Sharma et al, 2017) Rate (Albarqouni et al, 2016a), (Albarqouni et al, 2016b), (Brady et al, 2014), (Brady et al, 2017), , (dos Reis et al, 2015), (Eickhoff, 2014), (Foncubierta Rodríguez and Müller, 2012), (Gur et al, 2017), (de Herrera et al, 2014), (Holst et al, 2015), (Keshavan et al, 2018), (Mavandadi et al, 2012), (McKenna et al, 2012), (Mitry ...…”

“…Finally, we summarize the wages paid to crowd workers. (Eickhoff, 2014), (Gurari et al, 2015b), (Gurari et al, 2016), (Heim, 2018), (Irshad et al, 2015), (Maier-Hein et al, 2014a), (Maier-Hein et al, 2016), (McKenna et al, 2012), (Mitry et al, 2015), (Nguyen et al, 2012), (Ørting et al, 2017), (Park et al, 2016), (Park et al, 2017), (Park et al, 2018), (Sameki et al, 2016), (Irshad et al, 2017) (Boorboor et al, 2018), (Brady et al, 2014), (Cheplygina et al, 2016), (Della Mea et al, 2014), (Ganz et al, 2017), (Holst et al, 2015), (Lee and Tufail, 2014), (Lee et al, 2016), (Maier-Hein et al, 2014b), (Maier-Hein et al, 2015), (Mitry et al, 2013), (Mitry et al, 2016), (Sharma et al, (Albarqouni et al, 2016b), (Rajchl et al, 2016), (Smittenaar et al, 2018), (Sullivan et al, 2018), (Timmermans et al, 2016) (Albarqouni et al,...…”

A crowdsourcing semi-automatic image segmentation platform for cell biology

“…Recent studies on crowdsourcing have shown promising results on biomedical images. For example, [21] applied crowdsourcing techniques for the detection of dividing cells in breast cancer histology images, while [22] used a crowdsourcing framework for lung nodule detection and annotation to aid radiologists in lung cancer diagnosis.…”