Tests of Conditional Predictive Ability

Abstract

We propose a framework for out-of-sample predictive ability testing and forecast selection designed for use in the realistic situation in which the forecasting model is possibly misspecified, due to unmodeled dynamics, unmodeled heterogeneity, incorrect functional form, or any combination of these. Relative to the existing literature (Diebold and Mariano (1995) and West (1996)), we introduce two main innovations: (i) We derive our tests in an environment where the finite sample properties of the estimators on which the forecasts may depend are preserved asymptotically. (ii) We accommodate conditional evaluation objectives (can we predict which forecast will be more accurate at a future date?), which nest unconditional objectives (which forecast was more accurate on average?), that have been the sole focus of previous literature. As a result of (i), our tests have several advantages: they capture the effect of estimation uncertainty on relative forecast performance, they can handle forecasts based on both nested and nonnested models, they allow the forecasts to be produced by general estimation methods, and they are easy to compute. Although both unconditional and conditional approaches are informative, conditioning can help fine-tune the forecast selection to current economic conditions. To this end, we propose a two-step decision rule that uses current information to select the best forecast for the future date of interest. We illustrate the usefulness of our approach by comparing forecasts from leading parameter-reduction methods for macroeconomic forecasting using a large number of predictors. Copyright The Econometric Society 2006.

Deja Vol: Predictive Regressions for Aggregate Stock Market Volatility Using Macroeconomic Variables

“…For empirically realistic sample sizes, the simulation results illustrate a pattern of divergence between the Giacomini and White (2006) and Clark and West (2007) tests consistent with that observed in the data.…”

“…When the forecasts from the larger model are highly volatile relative to forecasts based on the benchmark (parsimonious) model, the additional noise related to parameter estimation is large, and the adjustment term in Eq. (5) Given two alternative forecasting methods, Giacomini and White (2006) propose a test for superior predictive ability. Let G t represent some set of information available at time t. Under the null hypothesis, the two models possess equal forecasting ability.…”

“…The general framework developed by Giacomini and White (2006) permits conditional comparison of forecast performance. Taking G t = { / 0, Ω}, the trivial information set, results in an unconditional test of equal predictive ability.…”

“…Both tests involve the out-ofsample difference in mean square prediction error (MSPE) relative to the benchmark. The first test, proposed by Giacomini and White (2006), is equivalent to the Diebold and Mariano (1995) test for equal predictive ability. The second test, proposed by Clark and West (2007), adds an adjustment term to the out-of-sample difference in MSPE that accounts for parameter estimation noise.…”

“…In the Clark and West (2007) framework, the null hypothesis involves the population difference in MSPE between the two nested models. By contrast, the null hypothesis in the Giacomini and White (2006) framework relates to the forecasting method and explicitly incorporates parameter estimation as a source of forecast error. The Clark and West (2007) test is appropriate when the underlying research question involves Granger causality, whereas the Giacomini and White (2006) test is appropriate for addressing the normatively oriented question of whether one forecast performs better than the other.…”