The wisdom of smaller, smarter crowds

Abstract

The "wisdom of crowds" refers to the phenomenon that aggregated predictions from a large group of people can rival or even beat the accuracy of experts. In domains with substantial stochastic elements, such as stock picking, crowd strategies (e.g. indexing) are difficult to beat. However, in domains in which some crowd members have demonstrably more skill than others, smart sub-crowds could possibly outperform the whole. The central question this work addresses is whether such smart subsets of a crowd can be identified a priori in a large-scale prediction contest that has substantial skill and luck components. We study this question with data obtained from fantasy soccer, a game in which millions of people choose professional players from the English Premier League to be on their fantasy soccer teams. The better the professional players do in real life games, the more points fantasy teams earn. Fantasy soccer is ideally suited to this investigation because it comprises millions of individual-level, within-subject predictions, past performance indicators, and the ability to test the effectiveness of arbitrary player-selection strategies. We find that smaller, smarter crowds can be identified in advance and that they beat the wisdom of the larger crowd. We also show that many players would do better by simply imitating the strategy of a player who has done well in the past. Finally, we provide a theoretical model that explains the results we see from our empirical analyses.

Rescuing Collective Wisdom when the Average Group Opinion Is Wrong

“…As the number of real-world domains where WOC methods have been applied increases, researchers are beginning to appreciate that each new domain requires considerable tuning of older methods in order to reach optimal performance. Early focus on universal simple strategies (Condorcet, 1785;Surowiecki, 2004;Hastie and Kameda, 2005) has been replaced with a plethora of methods that have sought to find a better match between the problem and the solution and by doing so have shown increases in performance relative to the averaging baseline (Goldstein et al, 2014;Budescu and Chen, 2015;Madirolas and de Polavieja, 2015;Whalen and Yeung, 2015).…”

“…One particular area where synergy might be achieved concerns finding a better integration between the methods of task allocation and consensus achievement (Brambilla et al, 2013). It will be interesting to see whether within-swarm task allocation methods could be combined with methods of consensus achievement to simultaneously encourage both diversity of expertise and cooperative action, similar to how crowd intelligence methods benefit from contextdependent reliance on experts (Zhou et al, 2002;Ward et al, 2011;Goldstein et al, 2014). Similar ideas have already borne fruit in the training of expert ensembles of neural networks (Zhou et al, 2002).…”

“…In another study (Goldstein et al, 2014), nearly 100,000 thousand online fantasy football players were ranked in order of past performance. The investigators then formed virtual random subgroups which varied in size and the amount of experts they contained.…”

Forecasts or fortune-telling: When are expert judgements of safety risk valid?

“…Goldstein (2014) examined the problem of smaller smarter groupings within a larger pool, by analysing the results of online fantasy soccer competitions. The experiment showed that increasing group size improved performance up to a certain point, after which further increase resulted in less accurate predictions.…”

Crowd Learning: Improving Online Decision Making Using Crowdsourced Data

“…While it is generally believed such crowdsourced information is useful, this is often argued heuristically. A recent study [Goldstein et al, 2014] shows that crowd wisdom should be selected carefully to reach its potential. This raises the interesting question on how to harness the power of crowdsourcing, and what type of learning algorithms can effectively utilize crowdsourced data in addition to one's direct observations.…”