The power of collective intelligence in a signal detection task

Abstract

A novel spatial display for a classic signal detection problem was tested in an experiment with 30 naive volunteers and 2 experts, measuring the signal detection properties of a spatially organised display compared with temporally organised display. The collective performance of committees was also measured. The spatial display was quantitatively and qualitatively superior, resulting in a 4.6% increase in correctness of classifications and a 30.5% decrease in miss percentages. Qualitatively, 83% of participants preferred the new display. The collective receiver operating characteristic performance of the top half of the best performing naive participants was superior to that of most individuals and all experts. However, in an analysis that examined the value added of both the improved spatial display and the collective intelligence approach, the results demonstrated that the bulk of performance gains can be attributed to the improved design, as compared to the value of collective action, by roughly 2:1.

Improving what is published: Toward an evidence-based framework for manuscript review.

“…Signal-detection research suggests that researchers should explore the possible benefits of using a larger number of reviewers to assess submitted manuscripts. Cummings and Quimby (2018) found that combining multiple novice raters’ judgments in a standard signal-detection task yielded more accurate classification decisions than were yielded by individual expert raters (a phenomenon they termed “the power of collective intelligence”). Consistent with these results, Herron (2012) found that the predictive validity of medical journal article evaluations provided by nonexpert readers actually exceeded that of expert readers when a large number of nonexpert ratings were obtained.…”