Intuitive Biases in Choice vs. Estimation: Implications for the Wisdom of Crowds

Simmons, Joseph P.; Nelson, Leif D.; Galak, Jeff; Frederick, Shane

doi:10.2139/ssrn.1553935

Cited by 9 publications

(4 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Critics of WOC have pointed to instances when the crowd's wisdom has failed to deliver accurate predictions because the aggregate estimate was largely distorted by systematic group bias or by a large number of uninformed judges (Simmons et al 2011). As an alternative to simply averaging individual judgments, researchers have proposed weighted models that favor better, wiser, more experienced judges in the crowd (e.g., Aspinall 2010, Wang et al 2011.…”

Section: Introductionmentioning

confidence: 99%

Identifying Expertise to Extract the Wisdom of Crowds

2015

View full text Add to dashboard Cite

S tatistical aggregation is often used to combine multiple opinions within a group. Such aggregates outperform individuals, including experts, in various prediction and estimation tasks. This result is attributed to the "wisdom of crowds." We seek to improve the quality of such aggregates by eliminating poorly performing individuals from the crowd. We propose a new measure of contribution to assess the judges' performance relative to the group and use positive contributors to build a weighting model for aggregating forecasts. In Study 1, we analyze 1,233 judges forecasting almost 200 current events to illustrate the superiority of our model over unweighted models and models weighted by measures of absolute performance. In Study 2, we replicate our findings by using economic forecasts from the European Central Bank and show how the method can be used to identify smaller crowds of the top positive contributors. We show that the model derives its power from identifying experts who consistently outperform the crowd.Data, as supplemental material, are available at http://dx.

show abstract

Section: Introductionmentioning

confidence: 99%

Identifying Expertise to Extract the Wisdom of Crowds

2015

View full text Add to dashboard Cite

show abstract

“…Each judgment can be modeled as consisting of two components: information and error [Surowiecki 2005;Simmons et al 2011]. Intuitively, if the judgments are unbiased and independent, the errors (deviations from the ground truth) will largely cancel out through averaging.…”

Section: Introductionmentioning

confidence: 99%

“…Diversity of experience of judges is thought to prevent "group think" phenomena [Surowiecki 2005]. Lastly, there should be no systematic bias in the judgments of the individuals (for example, each judge being off by a constant amount) as this can severely impact the accuracy of the aggregated estimate [Simmons et al 2011;Lorenz et al 2011;Muchnik et al 2013].…”

Section: Introductionmentioning

confidence: 99%

The wisdom of smaller, smarter crowds

Goldstein

McAfee

Suri

2014

Proceedings of the Fifteenth ACM Conference on Economics and Computation

View full text Add to dashboard Cite

The "wisdom of crowds" refers to the phenomenon that aggregated predictions from a large group of people can rival or even beat the accuracy of experts. In domains with substantial stochastic elements, such as stock picking, crowd strategies (e.g. indexing) are difficult to beat. However, in domains in which some crowd members have demonstrably more skill than others, smart sub-crowds could possibly outperform the whole. The central question this work addresses is whether such smart subsets of a crowd can be identified a priori in a large-scale prediction contest that has substantial skill and luck components. We study this question with data obtained from fantasy soccer, a game in which millions of people choose professional players from the English Premier League to be on their fantasy soccer teams. The better the professional players do in real life games, the more points fantasy teams earn. Fantasy soccer is ideally suited to this investigation because it comprises millions of individual-level, within-subject predictions, past performance indicators, and the ability to test the effectiveness of arbitrary player-selection strategies. We find that smaller, smarter crowds can be identified in advance and that they beat the wisdom of the larger crowd. We also show that many players would do better by simply imitating the strategy of a player who has done well in the past. Finally, we provide a theoretical model that explains the results we see from our empirical analyses.

show abstract

“…However, there is also evidence that the superiority of groups is not a given (Janis, 1982;Kerr, MacCoun, & Kramer, 1996;Simmons, Nelson, Galak, & Frederick, 2011). In many cases, groups have been shown to make disastrous decisions with deadly consequences (e.g., the Challenger explosion, Bay of Pigs incident, instances of ethnic cleansing, etc.…”

mentioning

confidence: 99%

Group Decision-Making

Tindale¹,

Winget²

2019

Oxford Research Encyclopedia of Psychology

View full text Add to dashboard Cite

Group decisions are ubiquitous in everyday life. Even when decisions are made individually, decision-makers often receive advice or suggestions from others. Thus, decisions are often social in nature and involve multiple group members. The literature on group decision-making is conceptualized as falling along two dimensions: how much interaction or information exchange is allowed among the group members, and how the final decision is made. On one end, group decisions can be made simply by aggregating member preferences or judgments without any interaction among members, with members having no control or say in the final judgment. One the other end, groups’ decisions can involve extensive member interaction and information exchanges, and the final decision is reached by group consensus. In between these two endpoints, various other strategies are also possible, including prediction markets, Delphi groups, and judge–advisor systems. Research has shown that each dimension has different implications for decision quality and process depending on the decision task and context. Research exploring these two dimension has also helped to illuminate those aspects of group decision-making that can lead to better-quality decisions.

show abstract

Intuitive Biases in Choice vs. Estimation: Implications for the Wisdom of Crowds

Cited by 9 publications

References 51 publications

Identifying Expertise to Extract the Wisdom of Crowds

Identifying Expertise to Extract the Wisdom of Crowds

The wisdom of smaller, smarter crowds

Group Decision-Making

Contact Info

Product

Resources

About