A hypothesis test algorithm for determining when weighting individual judgments reliably improves collective accuracy or just adds noise.

Abstract

The wisdom of a crowd can be extracted by simply averaging judgments, but weighting judges based on their past performance may improve accuracy. The reliability of any proposed weighting scheme depends on the estimation precision of the features that determine the weights, which in practice cannot be known perfectly. Therefore, we can never guarantee that any weighted average will be more accurate than the simple average. However, depending on the statistical properties of the judgments (i.e., their estimated biases, variances, and correlations) and the sample size (i.e., the number of judgments from each individual), we may be reasonably confident that a weighted average will outperform the simple average. We develop a general algorithm to test whether there are sufficiently many observed judgments for practitioners to reject using the simple average and instead trust a weighted average as a reliably more accurate judgment aggregation method. Using simulation, we find our test provides better guidance than cross validation. Using real data, we demonstrate how many judgments may be required to be able to trust commonly used weighted averages. Our algorithm can also be used for power analysis when planning data collection and as a decision tool given existing data to optimize crowd wisdom.

The Strengths and Weaknesses of Crowds to Address Global Problems

“…If influencers are creating distributions of tweets that differ from the true distribution of honest reactions, they can strongly influence the crowd’s judgment. Differential weighting, even when based on accurate information about validity, can often produce less accurate aggregates if there is uncertainty about what the weights should be (Huang et al, 2022). The problem of misinformation on Twitter is that weight may not even be based on validity at all, but on some other group-based criterion, such as fame, entertainment value, or a reaction that incidentally supports unpopular beliefs.…”

Introduction to the special issue on judgment and decision research on the wisdom of the crowds.

“…The ideas from the various articles can be combined to generate new research directions. For example, perhaps Huang et al’s (2024) hypothesis test can be applied to Hasan et al’s cancer classification task to determine whether the pathologists should be differentially weighted. And perhaps Beauchamp et al’s approach to modeling influence functions could be combined with Mayer and Heck’s work to determine the optimal order of judges in sequential collaboration (which itself may benefit from providing each collaborator multiple adjustments, to take advantage of the inner crowds as well as the external one).…”

“…The articles in this special issue on the wisdom of the crowds contribute to outstanding questions regarding what to aggregate (Feng & Budescu, 2024; Hasan et al, 2024; Summerville et al, 2024), optimal weighting schemes (Huang et al, 2024; Collins et al, 2024; Powell et al, 2024), incentives (Peker, 2024), establishing expertise (Howe et al, 2024), social processes (Beauchamp et al, 2024; Mayer & Heck, 2024), and beliefs about the efficacy of crowd wisdom (Schultze et al, 2024). One way to group the articles relates to an overarching question: To what extent should aggregation be left to the crowd members themselves?…”