The feasibility of using citizens to segment anatomy from medical images: Accuracy and motivation

Abstract

The development of automatic methods for segmenting anatomy from medical images is an important goal for many medical and healthcare research areas. Datasets that can be used to train and test computer algorithms, however, are often small due to the difficulties in obtaining experts to segment enough examples. Citizen science provides a potential solution to this problem but the feasibility of using the public to identify and segment anatomy in a medical image has not been investigated. Our study therefore aimed to explore the feasibility, in terms of performance and motivation, of using citizens for such purposes. Public involvement was woven into the study design and evaluation. Twenty-nine citizens were recruited and, after brief training, asked to segment the spine from a dataset of 150 magnetic resonance images. Participants segmented as many images as they could within three one-hour sessions. Their accuracy was evaluated by comparing them, as individuals and as a combined consensus, to the segmentations of three experts. Questionnaires and a focus group were used to determine the citizens’ motivation for taking part and their experience of the study. Citizen segmentation accuracy, in terms of agreement with the expert consensus segmentation, varied considerably between individual citizens. The citizen consensus, however, was close to the expert consensus, indicating that when pooled, citizens may be able to replace or supplement experts for generating large image datasets. Personal interest and a desire to help were the two most common reasons for taking part in the study.

An open innovation approach to co-produce scientific knowledge: an examination of citizen science in the healthcare ecosystem

“…Literature has associated manifold outcomes to citizen science. Sticking to a crowdsourcing approach to citizen science, it has been argued that lay people have a critical role in training machine learning algorithms to deal with big data (Meakin et al , 2019) and in supporting expert scientists to accomplish low value-added and time-spending data classification (Lee et al , 2017). Besides, drawing on a distributed-intelligence perspective, citizen scientists' involvement entails a greater awareness of health-related issues among lay people, which fosters the development of creative thinking (Dick, 2017) and paves the way for their engagement in participatory research activities (Kovacic et al , 2014).…”

Artificial intelligence in radiography: Where are we now and what does the future hold?

“…Supervised learning in medical imaging frequently uses image segmentation by experts to define anatomy and pathologies, but this technique is not limited to image interpretation or anatomical identification 9,11, 12 . AI tools are based on the accuracy of the ground truth training data and using a consensus of segmentations from multiple users has the benefit to increase accuracy, but costs more to develop the tool 13 . However, this means that supervised learning can be very expensive because experts are required to define the ground-truth images.…”