Randomness in scientific estimation is generally assumed to arise from unmeasured or uncontrolled factors. However, when combining subjective probability estimates, heterogeneity stemming from people's cognitive or information diversity is often more important than measurement noise. This paper presents a novel framework that uses partially overlapping information sources. A specific model is proposed within that framework and applied to the task of aggregating the probabilities given by a group of forecasters who predict whether an event will occur or not. Our model describes the distribution of information across forecasters in terms of easily interpretable parameters and shows how the optimal amount of extremizing of the average probability forecast (shifting it closer to its nearest extreme) varies as a function of the forecasters' information overlap. Our model thus gives a more principled understanding of the historically ad hoc practice of extremizing average forecasts. Supplementary material for this article is available online.

No abstract

How much can rational people really disagree? If we can understand the limits of such disagreement, can we remove noise by labeling excess disagreement as irrational and then construct a group belief based on everyone's rational beliefs? Based on this idea, “Regularized Aggregation of One-Off Probability Predictions” by Satopää proposes a Bayesian aggregator that requires no user intervention and can be computed efficiently even for a large number of one-off probability predictions. To illustrate, the aggregator is evaluated on predictions collected during a four-year forecasting tournament sponsored by the U.S. intelligence community. The aggregator improves the squared error (a.k.a., the Brier score) of simple averaging by around 20% and other commonly used aggregators by 10%−25%. This advantage stems almost exclusively from improved calibration. An R package called braggR implements the method and is available on CRAN.

No abstract