The wisdom of the coherent: Improving correspondence with coherence-weighted aggregation.

Abstract

Previous research shows that variation in coherence (i.e., degrees of respect for axioms of probability calculus), when used as a basis for performance-weighted aggregation, can improve the accuracy of probability judgments. However, many aspects of coherence-weighted aggregation remain a mystery, including both prescriptive issues (e.g., how best to use coherence measures) and theoretical issues (e.g., why coherenceweighted aggregation is effective). Using data from six experiments in two earlier studies (N = 58, N = 2,858) employing either general-knowledge or statistical information integration tasks, we addressed many of these issues. Of prescriptive relevance, we examined the effectiveness of coherence-weighted aggregation as a function of judgment elicitation method, group size, weighting function, and the bias of the function's tuning parameter. Of descriptive relevance, we propose that coherenceweighted aggregation can improve accuracy via two distinct, task-dependent routes: a causal route in which the bases for scoring accuracy depend on conformity to coherence principles (e.g., Bayesian information integration) and a diagnostic route in which coherence serves as a cue to correct knowledge. The findings provide support for the efficacy of both routes, but they also highlight why coherence weighting, especially the most biased forms, sometimes imposes costs to accuracy. We conclude by sketching a decision-theoretic approach to how aggregators can sensibly leverage the wisdom of the coherent within the crowd.

An ensemble method utilising multiple thinking styles that boosts the wisdom of the inner crowd effect

“…How can people make accurate estimates for these types of questions? One promising approach is that of 'the wisdom of the crowds' [1][2][3][4][5][6][7][8][9][10][11][12] . In other words, the average estimate of a crowd of individuals yields surprisingly accurate estimates.…”

Introduction to the special issue on judgment and decision research on the wisdom of the crowds.

“…The articles in this special issue on the wisdom of the crowds contribute to outstanding questions regarding what to aggregate (Feng & Budescu, 2024; Hasan et al, 2024; Summerville et al, 2024), optimal weighting schemes (Huang et al, 2024; Collins et al, 2024; Powell et al, 2024), incentives (Peker, 2024), establishing expertise (Howe et al, 2024), social processes (Beauchamp et al, 2024; Mayer & Heck, 2024), and beliefs about the efficacy of crowd wisdom (Schultze et al, 2024). One way to group the articles relates to an overarching question: To what extent should aggregation be left to the crowd members themselves?…”

“…Weights are also the focus of the work by Collins et al (2024) who take a close look at coherence-based weights (Karvetski et al, 2013). Coherence describes the degree to which one’s judgments follow the rules, or axioms, of probability theory: nonnegativity, unitarity, and additivity.…”