Are Some Forecasters Really Better Than Others?

Abstract

In any dataset with individual forecasts of economic variables, some forecasters will perform better than others. However, it is possible that these ex post differences reflect sampling variation and thus overstate the ex ante differences between forecasters. In this paper, we present a simple test of the null hypothesis that all forecasters in the US Survey of Professional Forecasters have equal ability. We construct a test statistic that reflects both the relative and absolute performance of the forecaster and use bootstrap techniques to compare the empirical results with the equivalents obtained under the null hypothesis of equal forecaster ability. Results suggests limited evidence for the idea that the best forecasters are actually innately better than others, though there is evidence that a relatively small group of forecasters perform very poorly.

Disagreement versus uncertainty: Evidence from distribution forecasts

“…5 In the rare cases that the estimated variance is negative, we resort to a Newey et al (1987) variance estimator using Bartlett weights, together with the Newey and West (1994) method for bandwidth selection, see Zeileis (2004) for implementation details in the R package ''sandwich''. advantage: The SPF data is a panel of changing composition, with frequent entry and exit of individual forecasters (Capistrán and Timmermann, 2009;Engelberg et al, 2011;D'Agostino et al, 2012). This renders estimating individual specific forecast variances or combination weights from past data very difficult.…”

All Forecasters Are Not the Same: Time-Varying Predictive Ability across Forecast Environments

“…Their analysis yields mixed results, with tests suggesting equal predictive performance for some variables and not others. D'Agostino, McQuinn, and Whelan (2012) point out that there are notable drawbacks to the approaches in these previous studies. The requirement of a balanced panel can significantly reduce sample size so that most of the information available from surveys is lost.…”

“…In addition, metrics based on a period-by-period summation of ranks provide limited information about the nature of participants' forecast errors on either an absolute or a relative basis. To remedy these shortcomings, D'Agostino, McQuinn, and Whelan (2012) develop a test for equal predictive performance that applies bootstrapping and Monte Carlo simulation techniques to a metric based on participants' squared forecast errors. Importantly, their approach recognizes that comparing forecasts within an unbalanced panel can be challenging because of time-variation in the forecasting environment.…”

All Forecasters Are Not the Same: Time-Varying Predictive Ability across Forecast Environments

“…The approach of D'Agostino, McQuinn, and Whelan (2012) begins by constructing a normalized squared error statistic for each variable for each period for each participant. Abstracting from details related to data and survey features, the normalized squared error statistic for participant j, ,…”

“…Importantly, the specification in (3) also nests two alternative approaches to control for changes in the forecastability of a variable. For example, the normalization procedure adopted by D'Agostino, McQuinn, Whelan (2012) and Meyler (2020) corresponds to the special case of the j α 's being jointly equal to zero. Alternatively, the application of time fixed effects corresponds to the case of equality of the j λ 's.…”