A universal method for evaluating the quality of aggregators

Abstract

We propose a new method to facilitate comparison of aggregated forecasts based on different aggregation, elicitation and calibration methods. Aggregates are evaluated by their relative position on the cumulative distribution of the corresponding individual scores. This allows one to compare methods using different measures of quality that use different scales. We illustrate the use of the method by re-analyzing various estimates from Budescu and Du (Management Science, 2007).

The wisdom of many in few: Finding individuals who are as wise as the crowd.

“…An important next step is to validate the utility of our findings in more practical or applied settings. One especially appealing direction would be to apply our methods on forecasting data with much longer time horizons, such as the Survey of Professional Forecasters data for economic indicators (Garcia, 2003; Han & Budescu, 2019, 2022; Himmelstein, Budescu & Han, 2022). The methods could also be validated on existing forecasting tournament data, such as from ACE or HFC, or other publicly available forecasting data.…”

“…Many forecasting studies rely on observational data (Han & Budescu, 2019, 2022Mandel & Barnes, 2014;Spann & Skiera, 2009), and typical forecasting tournaments allow judges to self-select the events and timing of their forecasts (Atanasov et al, 2017(Atanasov et al, , 2020Himmelstein et al, 2021;Morstatter et al, 2019). We implemented a new design where a large number of judges forecasted the same events repeatedly at the same times, eliminating many common confounds intrinsic to forecasting research, and allowing us to evaluate the forecasters at different time points.…”

“…Past research exploring the utility of this approach has proved fruitful but methodologically challenging. Most research on the psychology of forecasting utilizes one of two sources of data: (a) judgments collected during forecasting tournaments, in which large samples of forecasters self-select from among hundreds of possible forecasting questions, and make their predictions at different points in time, at their leisure (Atanasov & Himmelstein, 2022; Himmelstein et al, 2021; Mellers, Stone, Atanasov, et al, 2015; Morstatter et al, 2019; Tetlock & Gardner, 2016); (b) professional forecasters, who make predictions for a living about finance, sports, weather, health, politics, and more (Garcia, 2003; Han & Budescu, 2019, 2022; Himmelstein, Budescu & Han, 2022; Mandel & Barnes, 2014; Spann & Skiera, 2009). Both data sources inherently include several confounds that make controlled scientific inference difficult.…”

“…To evaluate the rules empirically, we conducted a novel longitudinal forecasting experiment specifically designed to experimentally eliminate common confounds that plague many forecasting datasets. Forecasting studies are often based on data collected during forecasting tournaments, such as the Aggregative Contingent Estimation (ACE) program (Atanasov et al, 2017(Atanasov et al, , 2020 and the Hybrid Forecasting Competition (HFC, Himmelstein et al, 2021;Morstatter et al, 2019); or in nonexperimental settings where professional forecasters predict economic, political, athletic, weather, health, and other types of events for a living (Garcia, 2003;Han & Budescu, 2019, 2022Mandel & Barnes, 2014;Spann & Skiera, 2009).…”

Recalibrating probabilistic forecasts to improve their accuracy

“…This function has some attractive properties: (1) it generates probabilities (and does not require any additional normalizations) for binary events, for any value of ; (2) = for three "natural" anchor points = 0, 0.5 and 1. The full LLO function and its simplified version (Equation 3) have been applied in a large body of studies and shown to enhance the accuracy of individual forecasts as well as aggregated forecasts (e.g., Atanasov et al, 2017;Budescu et al, 1997;Baron et al, 2014;Erev et al, 1994;Han & Budescu, 2019;Mellers et al, 2014;Satopää & Ungar, 2015;Shlomi & Wallsten, 2010;Turner et al, 2014). Mellers et al (2014) applied Karmarkar's transformation to data generated by more than 2,000 forecasters in a geopolitical forecasting tournament (Aggregative Contingent Estimation ACE; https://www.iarpa.gov/research-programs/ace).…”