Error measures for generalizing about forecasting methods: Empirical comparisons

Abstract

This study evaluated measures for making comparisons of errors across time series. We analyzed 90 annual and 101 quarterly economic time series. We judged error measures on reliability, construct validity, sensitivity to small changes, protection against outliers, and their relationship to decision making. The results lead us to recommend the Geometric Mean of the Relative Absolute Error (GMRAE) when the task involves calibrating a model for a set of time series. The GMRAE compares the absolute error of a given method to that from the random walk forecast. For selecting the most accurate methods, we recommend the Median RAE (MdRAE)when few series are available and the Median Absolute Percentage Error (MdAPE) otherwise. The Root Mean Square Error (RMSE) is not reliable, and is therefore inappropriate for comparing accuracy across series.

Good Night, and Good Luck: Perspectives on Luck in Management Scholarship

“…Empirical studies of forecasting accuracy show that predicting important business outcomes is challenging (Durand, 2003). The average absolute percentage error, i.e., (forecast -outcome) / forecast, in forecasts of macro-economic quantities (e.g., inflation, exchange rates, unemployment) by economists and analysts is about 20% (Armstrong & Collopy, 1992). Forecasts about demand and product success are even less accurate, with an average absolute percentage error of close to 50% (Fildes, Goodwin, Lawrence, & Nikolopoulos, 2009).…”

Professional vs. amateur judgment accuracy: The case of foreign exchange rates

“…One of the most commonly used measures of overall point judgment accuracy is the median absolute percentage error (MdAPE; cf., Armstrong & Collopy, 1992). Its 0-100 range is one of its primary attractions, for it permits easy comparisons across quantities that have radically different scales, as in the present study, e.g., US dollars vs. Japanese yen vs. Turkish lira.…”

Forecasting domestic liquidity during a crisis: what works best?

“…The advantage of the relative RMSE statistic is that it is independent of the scale of the variables. This method is preferred when comparing the utility of different forecasting models across data sets that have dissimilar scales (see Armstrong and Collopy, 1992).…”