When is a crowd wise?

Davis‐Stober, Clintin P.; Budescu, David V.; Dana, Jason; Broomell, Stephen B.

doi:10.1037/dec0000004

Cited by 155 publications

(173 citation statements)

References 37 publications

Supporting

Mentioning

157

Contrasting

Order By: Relevance

“…Averbeck et al (2006), for example, study the errors induced in decoding if neural activity correlations are ignored, and find that ignoring correlations generally decreases the performance of decoders when compared to the optimal decoder which takes the information present in correlations into account. Similar to Davis-Stober et al (2014) who study correlated opinions, they find a range of situations where correlations improve decoding accuracy as compared to independently activating neurons.…”

Section: Correlations and Continuous Variablesmentioning

confidence: 63%

“…Finding conditions where errors are independent helps speed up the decrease of variance. The most rapid decrease occurs when correlations are negative (Davis-Stober et al, 2014). For large negative correlations, the errors in pairs of individuals almost exactly cancel and even a very small group can function as well as a large crowd of independent individuals.…”

Section: Correlations and Continuous Variablesmentioning

confidence: 92%

See 1 more Smart Citation

Rescuing Collective Wisdom when the Average Group Opinion Is Wrong

2017

View full text Add to dashboard Cite

The total knowledge contained within a collective supersedes the knowledge of even its most intelligent member. Yet the collective knowledge will remain inaccessible to us unless we are able to find efficient knowledge aggregation methods that produce reliable decisions based on the behavior or opinions of the collective's members. It is often stated that simple averaging of a pool of opinions is a good and in many cases the optimal way to extract knowledge from a crowd. The method of averaging has been applied to analysis of decision-making in very different fields, such as forecasting, collective animal behavior, individual psychology, and machine learning. Two mathematical theorems, Condorcet's theorem and Jensen's inequality, provide a general theoretical justification for the averaging procedure. Yet the necessary conditions which guarantee the applicability of these theorems are often not met in practice. Under such circumstances, averaging can lead to suboptimal and sometimes very poor performance. Practitioners in many different fields have independently developed procedures to counteract the failures of averaging. We review such knowledge aggregation procedures and interpret the methods in the light of a statistical decision theory framework to explain when their application is justified. Our analysis indicates that in the ideal case, there should be a matching between the aggregation procedure and the nature of the knowledge distribution, correlations, and associated error costs. This leads us to explore how machine learning techniques can be used to extract near-optimal decision rules in a data-driven manner. We end with a discussion of open frontiers in the domain of knowledge aggregation and collective intelligence in general.

show abstract

Section: Correlations and Continuous Variablesmentioning

confidence: 63%

Section: Correlations and Continuous Variablesmentioning

confidence: 92%

Rescuing Collective Wisdom when the Average Group Opinion Is Wrong

2017

View full text Add to dashboard Cite

show abstract

“…Indeed, Larrick et al (2011) define the WOC effect as the fact that the average of the judges beats the average judge. Davis-Stober et al (2014) propose a more general definition, namely that some linear combination of the crowd's estimates should beat that of a randomly selected member of the crowd. Their analysis indicates that WOC is likely to be observed in a wide variety of situations with relatively few exceptions.…”

Section: Introductionmentioning

confidence: 99%

Identifying Expertise to Extract the Wisdom of Crowds

2015

Self Cite

View full text Add to dashboard Cite

S tatistical aggregation is often used to combine multiple opinions within a group. Such aggregates outperform individuals, including experts, in various prediction and estimation tasks. This result is attributed to the "wisdom of crowds." We seek to improve the quality of such aggregates by eliminating poorly performing individuals from the crowd. We propose a new measure of contribution to assess the judges' performance relative to the group and use positive contributors to build a weighting model for aggregating forecasts. In Study 1, we analyze 1,233 judges forecasting almost 200 current events to illustrate the superiority of our model over unweighted models and models weighted by measures of absolute performance. In Study 2, we replicate our findings by using economic forecasts from the European Central Bank and show how the method can be used to identify smaller crowds of the top positive contributors. We show that the model derives its power from identifying experts who consistently outperform the crowd.Data, as supplemental material, are available at http://dx.

show abstract

“…Since the actual group performance was inferior to the average model descriptively, we can conclude that to the extent that groups engaged in differential weighting, they did not benefit from it in Experiment 1. In contrast, groups engaged in effective differential weighting in Experiment 2, which employed a task favoring weighting by expertise or accuracy over averaging, due to a strong population bias (e.g., Davis-Stober, Budescu, Dana, & Broomell, 2014;Einhorn et al, 1977). This allowed interacting groups in Experiment 2 to outperform the simple average of their members' individual estimates.…”

Section: Superiority Of Group Judgments and Differential Weightingmentioning

confidence: 99%

How Much Group is Necessary? Group-To-Individual Transfer in Estimation Tasks

Stern¹,

Schultze²,

Schulz‐Hardt³

2017

Collabra: Psychology

View full text Add to dashboard Cite

Hardt G-I transfer denotes an increase in individual performance due to group interaction, for example, because of acquiring certain skills or knowledge from the other group members. Whereas such G-I transfer has been successfully shown for problem-solving tasks, evidence for G-I transfer on quantitative estimation tasks is scarce. We address this research gap with a focus on how often a group has to interact in order to fully exploit the benefit of this learning effect. Results from two experiments support the idea that a single group interaction is sufficient to induce a stable G-I transfer, which reduces group members' metric error. Smaller metric errors indicate that people improved their representation of the correct upper and lower boundaries, or what range of values is plausible. In contrast to nominal groups, both members of continuously interacting groups and members of groups with only one initial interaction exhibited stable G-I transfer, and the size of this transfer did not significantly differ between the latter two conditions. Furthermore, we found evidence for differential weighting of group members' individual contributions that goes beyond sheer individual capability gains under certain circumstances, namely in tasks with a population bias.

show abstract

When is a crowd wise?

Cited by 155 publications

References 37 publications

Rescuing Collective Wisdom when the Average Group Opinion Is Wrong

Rescuing Collective Wisdom when the Average Group Opinion Is Wrong

Identifying Expertise to Extract the Wisdom of Crowds

How Much Group is Necessary? Group-To-Individual Transfer in Estimation Tasks

Contact Info

Product

Resources

About