Evaluating Forecasts of a Vector of Variables: A German Forecasting Competition

Abstract

In this paper we explore methodologies appropriate for evaluating a forecasting competition when the participants predict a number of variables that may be related to each other and are judged for a single period. Typically, forecasting competitions are judged on a variable-by-variable basis, but a multivariate analysis is required to determine how each competitor performed overall. We use three different multivariate tests to determine an overall winner for a forecasting competition for the German economy across 25 different institutions for a single time period using a vector of eight key economic variables. We find that neglecting the cross-variable relationships greatly alters the outcome of the forecasting competition. Davies et al. (2011) provide a useful summary of the framework used in these papers. 6 Clements and Hendry (1993) did consider the methodology of evaluating forecasts from dynamic systems. 494 T. Sinclair et al.

A test of the joint efficiency of macroeconomic forecasts using multivariate random forests

“…The Mahalanobis distance is a generalization of the Euclidian distance that captures a nonzero correlation of the forecast errors (and different scales; see, e.g., Sinclair et al, 2016; for the case of a region-specific covariance matrix, see Segal, 1992). When V is the identity matrix, the Mahalanobis distance is equal to the Euclidean distance.…”

“…In order to add further economic intuition to our analysis, we compute a joint-forecast-efficiency ranking for the four research institutes (for a ranking of joint forecast accuracy based on the Mahalanobis distance, see Sinclair et al, 2016). To this end, we sum up the t-statistics that we have found in the three scenarios considered in Tables II-IV. A large negative t-statistic indicates strong rejection of joint forecast efficiency.…”

“…When V is the identity matrix, the Mahalanobis distance is equal to the Euclidean distance. The Mahalanobis distance is a generalization of the Euclidian distance that captures a nonzero correlation of the forecast errors (and different scales; see, e.g., Sinclair et al, 2016; for the case of a region-specific covariance matrix, see Segal, 1992).…”

“…In addition to contributing to the recent literature on multivariate forecast evaluation, our research also contributes to the large and significant literature on the properties of macroeconomic forecasts for Germany (for early contributions, see, among others, Kirchgässner, 1984;Neumann & Buscher, 1980). Kirchgässner (1993) and Sinclair et al (2016) combine both literatures and use a multivariate setting to rank several German research institutions (see also Döhrn, 2015). In a univariate setting, Behrens, Pierdzioch, and Risse (2018a) use Bayesian regression trees to analyze the efficiency of growth and inflation forecasts.…”

“…Sinclair and Stekler (2013) use this approach, which combines a VAR model and the Mahalanobis distance, to assess the quality of initial estimates of macroeconomic variables, and Sinclair, Stekler, and Carnow (2015) use the same approach to study the bias and accuracy of a vector of forecasts published by the Federal Reserve. Other researchers have used the Mahalanobis distance as a metric of multivariate forecast dispersion (Banternghansa & McCracken, 2009) and as a metric to rank forecasters in a multivariate setting (Bauer, Eisenbeis, Waggoner, & Zha, 2003;Eisenbeis, Waggoner, & Zha, 2002;Sinclair, Stekler, & Muller-Droge, 2016). Yet other researchers have developed techniques that render it possible to test for the rationality of forecasts in a multivariate setting when a forecaster's loss function is nonseparable across variables (Kommunjer &Owyang, 2012, andKrüger, 2014, apply this technique to study the forecasts of the German Council of Economic Experts).…”