The Wisdom of Small Crowds

Mannes, Albert E.; Soll, Jack B.; Larrick, Rick

doi:10.1037/e509992015-080

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Afterwards the experimenter aggregates different size groups of these estimates and compares them to the best estimate and the aggregate of the entire group. Mannes et al [2013] recently and independently revisited the same idea. They analyze two data sets of numerical estimates.…”

Section: Related Workmentioning

confidence: 91%

See 1 more Smart Citation

The wisdom of smaller, smarter crowds

Goldstein

McAfee

Suri

2014

Proceedings of the Fifteenth ACM Conference on Economics and Computation

View full text Add to dashboard Cite

The "wisdom of crowds" refers to the phenomenon that aggregated predictions from a large group of people can rival or even beat the accuracy of experts. In domains with substantial stochastic elements, such as stock picking, crowd strategies (e.g. indexing) are difficult to beat. However, in domains in which some crowd members have demonstrably more skill than others, smart sub-crowds could possibly outperform the whole. The central question this work addresses is whether such smart subsets of a crowd can be identified a priori in a large-scale prediction contest that has substantial skill and luck components. We study this question with data obtained from fantasy soccer, a game in which millions of people choose professional players from the English Premier League to be on their fantasy soccer teams. The better the professional players do in real life games, the more points fantasy teams earn. Fantasy soccer is ideally suited to this investigation because it comprises millions of individual-level, within-subject predictions, past performance indicators, and the ability to test the effectiveness of arbitrary player-selection strategies. We find that smaller, smarter crowds can be identified in advance and that they beat the wisdom of the larger crowd. We also show that many players would do better by simply imitating the strategy of a player who has done well in the past. Finally, we provide a theoretical model that explains the results we see from our empirical analyses.

show abstract

Section: Related Workmentioning

confidence: 91%

“…Most of the prior works [Bruce 1936;Gordon 1924;Klugman 1945;Knight 1921;Lorge et al 1958;Mannes et al 2013] analyze data sets carefully designed to experimentally test the existence of small, smart crowds. The internet age has made it much easier to find data sets that are logs of some type of human behavior.…”

Section: Related Workmentioning

confidence: 99%

The wisdom of smaller, smarter crowds

Goldstein

McAfee

Suri

2014

Proceedings of the Fifteenth ACM Conference on Economics and Computation

View full text Add to dashboard Cite

show abstract

“…However, in practice, because the workers' prediction model and reliabilities are never known perfectly, we run the risk of adding noisy information from the unreliable workers as we include all the workers. In fact, a recent empirical study (Mannes, Soll, and Larrick 2013) showed that the aggregated results of a small number of (3 to 6) highquality workers are often more accurate than those of much larger crowds (a.k.a. the wisdom of small crowds).…”

Section: Introductionmentioning

confidence: 99%

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Liu

2015

HCOMP

View full text Add to dashboard Cite

Crowdsourcing provides a popular paradigm for data collection at scale. We study the problem of selecting subsets of workers from a given worker pool to maximize the accuracy under a budget constraint. One natural question is whether we should hire as many workers as the budget allows, or restrict on a small number of top-quality workers. By theoretically analyzing the error rate of a typical setting in crowdsourcing, we frame the worker selection problem into a combinatorial optimization problem and propose an algorithm to solve it efficiently. Empirical results on both simulated and real-world datasets show that our algorithm is able to select a small number of high-quality workers, and performs as good as, sometimes even better than, the much larger crowds as the budget allows.

show abstract

“…Even if there exist no extreme spammers, the median-level workers may still decrease the overall accuracy if they dominate over the small number of high-quality workers. In fact, a recent empirical study (Mannes et al, 2013) shows that the aggregated results of a small number of (3 to 6) high-quality workers are often more accurate than those of much larger crowds.…”

Section: Introductionmentioning

confidence: 99%

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Li¹,

Liu²

2015

Preprint

View full text Add to dashboard Cite

Crowdsourcing provides a popular paradigm for data collection at scale. We study the problem of selecting subsets of workers from a given worker pool to maximize the accuracy under a budget constraint. One natural question is whether we should hire as many workers as the budget allows, or restrict on a small number of topquality workers. By theoretically analyzing the error rate of a typical setting in crowdsourcing, we frame the worker selection problem into a combinatorial optimization problem and propose an algorithm to solve it efficiently. Empirical results on both simulated and real-world datasets show that our algorithm is able to select a small number of high-quality workers, and performs as good as, sometimes even better than, the much larger crowds as the budget allows.

show abstract

The Wisdom of Small Crowds

Cited by 3 publications

References 0 publications

The wisdom of smaller, smarter crowds

The wisdom of smaller, smarter crowds

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Contact Info

Product

Resources

About