On the limitations of comparing mean square forecast errors

Abstract

Linear models are invariant under non-singular, scale-preserving linear transformations, whereas mean square forecast errors (MSFEs) are not. Different rankings may result across models or methods from choosing alternative yet isomorphic representations of a process. One approach can dominate others for comparisons in levels, yet lose to another for differences, to a second for cointegrating vectors and to a third for combinations of variables. The potential for switches in ranking is related to criticisms of the inadequacy of MSFE against encompassing criteria, which are invariant under linear transforms and entail MSFE dominance. An invariant evaluation criterion which avoids misleading outcomes is examined in a Monte Carlo study of forecasting methods.

Path forecast evaluation

“…A natural way to overcome this difficulty is to construct a metric that evaluates entire forecast paths jointly. Clements and Hendry (1993) suggest to base comparison on the determinant of the forecast error second moment matrix pooled across horizons of interest, which they call generalized forecast error second moment (GFESM ). Compared to the standard MSFE, the GFESM has the advantage of being invariant to non-singular, scalepreserving linear transformations.…”

Testing for Differences in Path Forecast Accuracy: Forecast-Error Dynamics Matter

“…In order to evaluate the path forecast errors, we can use the general loss function proposed by Clements and Hendry (1993). Their general matrix of the forecast-error second-moment (GMFESM) extends the MSE matrix in (2.2) to multiple horizons and is computed as the mean squared path forecast errors…”

“…Assessments of path forecast accuracy cannot rely on traditional point accuracy metrics. Instead, we show that the general forecast-error second-moment (hereafter GFESM), proposed by Clements and Hendry (1993), can be reinterpreted as a metric of path forecast accuracy. The GFESM is effectively a multi-horizon generalization of the log determinant measure proposed by Doan et al (1984).…”

A vector error-correction forecasting model of the US economy

“…As recently as Stock (1995), the apparent value of incorporating cointegration into the forecasting exercise was noted. In a recent applications Clements and Hendry (1995) and Hoffman and Rasche (1996b) re-examine this issue, concluding that it is difficult to verify the predictions of Engle and Yoo in practice. Clements and Hendry (1995) find that incorporating knowledge of cointegration rank results in significant mean squared forecast error (MSFE) reduction only in models formed from relatively small samples.…”

“…Papers by Clements and Hendry (1993) and Hoffman and Rasche (1996b) employ measures of system performance, while Clements and Hendry (1993) and Christofferson and Diebold (1996) argue that conventional RMSE criterion may not capture some of the advantages of long-run information into the system. The basic conclusion of this body of literature is that incorporating cointegration may improve forecast performance, but improvement need not show up only at longer horizons as predicted originally by Engle and Yoo (1987).…”