Forecast evaluation often compares a parsimonious null model to a larger model that nests the null model. Under the null that the parsimonious model generates the data, the larger model introduces noise into its forecasts by estimating parameters whose population values are zero. We observe that the mean squared prediction error (MSPE) from the parsimonious model is therefore expected to be smaller than that of the larger model. We describe how to adjust MSPEs to account for this noise. We propose applying standard methods [West, K.D., 1996. Asymptotic inference about predictive ability. Econometrica 64, 1067-1084] to test whether the adjusted mean squared error difference is zero. We refer to nonstandard limiting distributions derived in Clark and McCracken [2001. Tests of equal forecast accuracy and encompassing for nested models. Journal of Econometrics 105, 85-110; 2005a. Evaluating direct multistep forecasts. Econometric Reviews 24, 369-404] to argue that use of standard normal critical values will yield actual sizes close to, but a little less than, nominal size. Simulation evidence supports our recommended procedure. r

This paper examines the asymptotic and finite-sample properties of out-of-sample tests for equal accuracy and encompassing as applied to nested models. With nested models, these tests can be viewed as Granger causality tests. Applied to nested models, however, the standard asymptotic critical values for many tests of equal accuracy and encompassing are invalid. Statistics such as those proposed by Diebold and Mariano (1995) and Harvey, et. al. (1998) fail to converge to the standard normal distribution when the models are nested rather than nonnested. Building on McCracken's (1999) results for equal accuracy tests, this paper derives the asymptotic distributions for a set of standard encompassing tests and one new encompassing test. Numerical simulations are used to generate the appropriate asymptotic critical values. Monte Carlo simulations are then used to evaluate the size and power of a battery of equal forecast accuracy and encompassing tests, as well as standard F-tests of causality. In these experiments, forecasts from an estimated VAR model are compared to those from a null estimated AR model. The simulation results indicate that McCracken's out-of-sample F-type test of equal accuracy and the encompassing test proposed in this paper can be more powerful than standard F-tests of causality. The Monte Carlo simulations also show that using invalid asymptotic critical values can produce misleading inferences.

West thanks the National Science Foundation for financial support. We thank Pablo M. Pincheira-Brown, Philip Hans Franses, Taisuke Nakata, Norm Swanson, participants in a session at the January 2006 meeting of the Econometric Society and two anonymous referees for helpful comments. The views expressed herein are solely those of the authors and do not necessarily reflect the views of the Federal Reserve Bank of Kansas City or the Federal Reserve System. The views expressed herein are those of the author(s) and do not necessarily reflect the views of the National Bureau of Economic Research.

This paper examines the asymptotic and finite-sample properties of out-of-sample tests for equal accuracy and encompassing as applied to nested models. With nested models, these tests can be viewed as Granger causality tests. Applied to nested models, however, the standard asymptotic critical values for many tests of equal accuracy and encompassing are invalid. Statistics such as those proposed by Diebold and Mariano (1995) and Harvey, et. al. (1998) fail to converge to the standard normal distribution when the models are nested rather than nonnested. Building on McCracken's (1999) results for equal accuracy tests, this paper derives the asymptotic distributions for a set of standard encompassing tests and one new encompassing test. Numerical simulations are used to generate the appropriate asymptotic critical values. Monte Carlo simulations are then used to evaluate the size and power of a battery of equal forecast accuracy and encompassing tests, as well as standard F-tests of causality. In these experiments, forecasts from an estimated VAR model are compared to those from a null estimated AR model. The simulation results indicate that McCracken's out-of-sample F-type test of equal accuracy and the encompassing test proposed in this paper can be more powerful than standard F-tests of causality. The Monte Carlo simulations also show that using invalid asymptotic critical values can produce misleading inferences.

SummaryThis paper compares alternative models of time‐varying volatility on the basis of the accuracy of real‐time point and density forecasts of key macroeconomic time series for the USA. We consider Bayesian autoregressive and vector autoregressive models that incorporate some form of time‐varying volatility, precisely random walk stochastic volatility, stochastic volatility following a stationary AR process, stochastic volatility coupled with fat tails, GARCH and mixture of innovation models. The results show that the AR and VAR specifications with conventional stochastic volatility dominate other volatility specifications, in terms of point forecasting to some degree and density forecasting to a greater degree. Copyright © 2014 John Wiley & Sons, Ltd.