Boosting medical diagnostics by pooling independent judgments

Abstract

Collective intelligence refers to the ability of groups to outperform individual decision makers when solving complex cognitive problems. Despite its potential to revolutionize decision making in a wide range of domains, including medical, economic, and political decision making, at present, little is known about the conditions underlying collective intelligence in real-world contexts. We here focus on two key areas of medical diagnostics, breast and skin cancer detection. Using a simulation study that draws on large real-world datasets, involving more than 140 doctors making more than 20,000 diagnoses, we investigate when combining the independent judgments of multiple doctors outperforms the best doctor in a group. We find that similarity in diagnostic accuracy is a key condition for collective intelligence: Aggregating the independent judgments of doctors outperforms the best doctor in a group whenever the diagnostic accuracy of doctors is relatively similar, but not when doctors' diagnostic accuracy differs too much. This intriguingly simple result is highly robust and holds across different group sizes, performance levels of the best doctor, and collective intelligence rules. The enabling role of similarity, in turn, is explained by its systematic effects on the number of correct and incorrect decisions of the best doctor that are overruled by the collective. By identifying a key factor underlying collective intelligence in two important real-world contexts, our findings pave the way for innovative and more effective approaches to complex real-world decision making, and to the scientific analyses of those approaches.collective intelligence | groups | medical diagnostics | dermatology | mammography C ollective intelligence, that is, the ability of groups to outperform individual decision makers when solving complex cognitive problems, is a powerful approach for boosting decision accuracy (1-7). However, despite its potential to boost accuracy in a wide range of contexts, including lie detection, political forecasting, investment decisions, and medical decision making (8-14), little is known about the conditions that underlie the emergence of collective intelligence in real-world domains. Which features of decision makers and decision contexts favor the emergence of collective intelligence? Which decision-making rules permit this potential to be harnessed? We here provide answers to these important questions in the domain of medical diagnostics.Our work builds on recent findings on combining decisions, a research paradigm known as "two heads better than one" (15)(16)(17)(18)(19)(20). In their seminal study, Bahrami et al. (15) showed that two individuals permitted to communicate freely while engaging in a visual perception task, achieved better results than the better of the two did alone. Koriat (17) subsequently demonstrated that this collective intelligence effect also emerges in the absence of communication when the "maximum-confidence slating algorithm" (hereafter called confidence rule) is used and th...

Rescuing Collective Wisdom when the Average Group Opinion Is Wrong

“…We have also examined whether neural networks could improve upon the performance of previously proposed heuristics on a skin cancer classification dataset (Kurvers et al, 2016). In the dataset, forty doctors had given their estimations and subjective confidence scores (four point scale) on whether particular patients had malignant melanoma by examining images of their skin lesions.…”

“…In the dataset, forty doctors had given their estimations and subjective confidence scores (four point scale) on whether particular patients had malignant melanoma by examining images of their skin lesions. As in Kurvers et al (2016), we used Youden's index as a measure of accuracy, given by J = sensitivity + specificity − 1, with sensitivity defined as the proportion of positive cases correctly evaluated and specificity defined as the proportion of negative cases correctly evaluated. This measure weights equally sensitivity and specificity and it is, thus, insensitive to the unbalances of a dataset (in this case, more cases without cancer than with cancer).…”

The wisdom of crowds for visual search

“…Many studies have shown benefits in combining expert judgments and/ or crowdsourcing nonexpert judgments for a variety of disciplines, including radiology (41,42,44), ophthalmology (73,74), pathology (44,75,76), and clinical predictions (52). However, few studies (41,42,77) have compared the benefits of various pooling algorithms with those predicted by signal detection theory.…”

“…Although some groups seem to have leaders who make decisions alone on behalf of their groups (17,(21)(22)(23), it is difficult for individuals to outperform even simple aggregations of the entire group's individual judgments (4,7,9,10,19,(24)(25)(26). Perhaps that is why humans often make important decisions as a group (27-29), even if the only expedient (30, 31) but effective (24, 31-34) group decision mechanism is to use the simple majority voting rule (35).Previous human studies have shown that combining people's judgments into group decisions can lead to accuracy benefits in various domains, such as estimation (36-38), detection (34,(39)(40)(41)(42)(43)(44), identification (45-47), and prediction (46, 48-52), a phenomenon known as the wisdom of crowds (53). For artificial tasks, where perceptual decisions are limited only by noise that is internal to each observer's brain (i.e., no external noise), the maximum wisdom of crowd benefits are specified by the idealized signal detection theory model that treats observers' internal judgments as normally distributed and statistically independent (SDT-IND) (54).…”

“…Previous human studies have shown that combining people's judgments into group decisions can lead to accuracy benefits in various domains, such as estimation (36)(37)(38), detection (34,(39)(40)(41)(42)(43)(44), identification (45)(46)(47), and prediction (46,(48)(49)(50)(51)(52), a phenomenon known as the wisdom of crowds (53). For artificial tasks, where perceptual decisions are limited only by noise that is internal to each observer's brain (i.e., no external noise), the maximum wisdom of crowd benefits are specified by the idealized signal detection theory model that treats observers' internal judgments as normally distributed and statistically independent (SDT-IND) (54).…”