Evaluating the Calibration of Multi-Step-Ahead Density Forecasts Using Raw Moments

Abstract

Abstract:The evaluation of multi-step-ahead density forecasts is complicated by the serial correlation of the corresponding probability integral transforms. In the literature, three testing approaches can be found which take this problem into account.However, these approaches can be computationally burdensome, ignore important information and therefore lack power, or suffer from size distortions even asymptotically. In this work, a fourth testing approach based on raw moments is proposed.It is easy to implement, uses standard critical values, can include all moments regarded as important, and has correct asymptotic size. It is found to have good size and power properties if it is based directly on the (standardized) probability integral transforms.Keywords: Density forecast evaluation; normality tests JEL-Classification: C12, C52, C53 Non-technical SummaryToday, predictions are often made in the form of density forecasts. An increasing number of central banks publishes density forecasts, which are displayed by fan charts. Compared to point forecasts, density forecasts contain additional information. From density forecasts for inflation, for example, it is possible to infer the probability of deflation or the probability of inflation being higher than the central bank's target.Point forecasts can be evaluated according to properties like bias or efficiency.Similarly, density forecasts can also be evaluated. A forecast density should coincide with the true density of the variable under study. If this is the case, the density forecast is said to be correctly calibrated. However, if, for instance, over a certain period of time the realizations of the variable under study always occur within a very narrow interval around the means of the forecast densities, this would be a strong indication for incorrect calibration. The forecast densities probably have a too large width in this case.If density forecasts for more than one period ahead are to be evaluated, this evaluation is complicated by the serial correlation of the outcomes with respect to the density forecasts. Suppose, for example, that one density forecast for inflation is made in January for July, and the next forecast is made in February for August.Then, if inflation in July turns out to be much higher than the mean of January's density forecast, it is very likely that inflation in August will also be considerably higher than the mean of February's density forecast.One can distinguish three evaluation approaches that are used or suggested in the literature for these situations. However, each of them has certain disadvantages with respect to the ability to detect incorrect calibration, to the possibility of falsely concluding that the density forecasts have incorrect calibration although it is actually correct, or to the ease of use. Therefore, an alternative evaluation approach, which does not suffer from any of these drawbacks, is suggested in this paper. In simulations, this new approach is found to yield good results and, thus, to be a viable alternative t...

The Pricing Kernel Puzzle in Forward Looking Data

“…We confirm this finding by running the Knüppel (2015) test with the first two moments (L = 2) only and find that monotonicity cannot be rejected as well. This holds true irrespective of taking care of serial correlation by a proper covariance matrix Ω L or focusing on the moments only; i.e.…”

“…Compared to the base setting, only the Knüppel (2015) test now fails to reject monotonicity with a p-value of 19% in the risk-neutral simulation setting.…”

“…The Knüppel (2015) test fits the empirical moments of the uniform variables u t to their theoretical counterpart in a generalized method of moments (GMM) fashion. When considering L moments, the test statistic is given by…”

“…In our base procedure, following Knüppel (2015), we use the first four moments (L = 4) and set those elements of the covariance matrix to zero, which correspond to covariances between odd and even moments. We account for possible time dependence in the u t by using the Newey-West covariance matrix with the automatic lag length selection proposed by Andrews (1991).…”

“…Third, we provide a formal test of pricing kernel monotonicity by simulating the difference in average log scores between restricted and unrestricted pricing kernels. We extend our test to a number of alternative test statistics measuring the quality of the pricing kernel to explain the data, namely, the moment fit criterion presented in Knüppel (2015), the Cramér van Mises statistic, and the Berkowitz (2001) test.…”