Collective Intelligence Meets Medical Decision-Making: The Collective Outperforms the Best Radiologist

Abstract

While collective intelligence (CI) is a powerful approach to increase decision accuracy, few attempts have been made to unlock its potential in medical decision-making. Here we investigated the performance of three well-known collective intelligence rules (“majority”, “quorum”, and “weighted quorum”) when applied to mammography screening. For any particular mammogram, these rules aggregate the independent assessments of multiple radiologists into a single decision (recall the patient for additional workup or not). We found that, compared to single radiologists, any of these CI-rules both increases true positives (i.e., recalls of patients with cancer) and decreases false positives (i.e., recalls of patients without cancer), thereby overcoming one of the fundamental limitations to decision accuracy that individual radiologists face. Importantly, we find that all CI-rules systematically outperform even the best-performing individual radiologist in the respective group. Our findings demonstrate that CI can be employed to improve mammography screening; similarly, CI may have the potential to improve medical decision-making in a much wider range of contexts, including many areas of diagnostic imaging and, more generally, diagnostic decisions that are based on the subjective interpretation of evidence.

Comparative Accuracy of Diagnosis by Collective Intelligence of Multiple Physicians vs Individual Physicians

“…Diagnostic accuracy was consistently greater with increasing group size, similar to findings of earlier literature on medical student teams 11,16 and combinations of user opinions for image interpretation. 8,9,10 These findings held across a broad range of specifications for creating a collective differential. These findings suggest that using the concept of collective intelligence to pool many physicians’ diagnoses could be a scalable approach to improve diagnostic accuracy.…”

“…5 The ability of collective intelligence—groups of individuals acting independently or collectively—to outperform individuals acting alone has gained prominence in politics, business, and economics, with studies identifying conditions in which collective intelligence appears beneficial. 8,9,10,11,12,13,14,15 Collective intelligence facilitated by software is particularly well suited for medicine because it may offer superior performance with little coordination.…”

Boosting medical diagnostics by pooling independent judgments

“…However, despite its potential to boost accuracy in a wide range of contexts, including lie detection, political forecasting, investment decisions, and medical decision making (8)(9)(10)(11)(12)(13)(14), little is known about the conditions that underlie the emergence of collective intelligence in real-world domains. Which features of decision makers and decision contexts favor the emergence of collective intelligence?…”

“…Reducing the frequency of diagnostic errors is thus a major step toward improving health care (24,25). Previous research on collective intelligence in medical diagnostics has yielded conflicting results: Some studies have found that group decision making boosts diagnostic accuracy (9,12,26,27), whereas others have found null or even detrimental effects (28,29).…”

Collaboration between a human group and artificial intelligence can improve prediction of multiple sclerosis course: a proof-of-principle study

“…In principle, the larger the group, the higher the prediction accuracy (see for review Ponsonby & Mattingly, 2015), which led to the development of several crowdsourcing initiatives for diagnostic purposes (for instance, Candido dos Reis et al , 2015; Lau et al , 2016). However, when expert people are involved, even small groups can outperform the best among them, at least when a yes/no answer to well-defined diagnostic questions is requested based on radiographic/ histological images, ( Kurvers et al , 2016; Sonabend et al , 2017; Wolf et al , 2015). Studies with medical students show that working in pairs ameliorates diagnostic ability, with further improvements when group size increases ( Hautz et al , 2015; Kämmer et al , 2017), in line with the core idea of Collective intelligence.…”