An application of rule-based forecasting to a situation lacking domain knowledge

Abstract

Rule-based forecasting (RBF) uses rules to combine forecasts from simple extrapolation methods. Weights for combining the rules use statistical and domain-based features of time series. RBF was originally developed, tested, and validated only on annual data. For the M3-Competition, three major modifications were made to RBF. First, due to the absence of much in the way of domain knowledge, we prepared the forecasts under the assumption that no domain knowledge was available. This removes what we believe is one of RBF's primary advantages. We had to re-calibrate some of the rules relating to causal forces to allow for this lack of domain knowledge. Second, automatic identification procedures were used for six time-series features that had previously been identified using judgment. This was done to reduce cost and improve reliability. Third, we simplified the rule-base by removing one method from the four that were used in the original implementation. Although this resulted in some loss in accuracy, it reduced the number of rules in the rule-base from 99 to 64. This version of RBF still benefits from the use of prior findings on extrapolation, so we expected that it would be substantially more accurate than the random walk and somewhat more accurate than equal weights combining. Because most of the previous work on RBF was done using annual data, we especially expected it to perform well with annual data. AbstractRule-based forecasting (RBF) uses rules to combine forecasts from simple extrapolation methods. Weights for combining the rules use statistical and domain-based features of time series. RBF was originally developed, tested, and validated only on annual data. For the M3-Competition, three major modifications were made to RBF. First, due to the absence of much in the way of domain knowledge, we prepared the forecasts under the assumption that no domain knowledge was available. This removes what we believe is one of RBF's primary advantages. We had to re-calibrate some of the rules relating to causal forces to allow for this lack of domain knowledge. Second, automatic identification procedures were used for six time-series features that had previously been identified using judgment. This was done to reduce cost and improve reliability. Third, we simplified the rule-base by removing one method from the four that were used in the original implementation. Although this resulted in some loss in accuracy, it reduced the number of rules in the rule-base from 99 to 64. This version of RBF still benefits from the use of prior findings on extrapolation, so we expected that it would be substantially more accurate than the random walk and somewhat more accurate than equal weights combining. Because most of the previous work on RBF was done using annual data, we especially expected it to perform well with annual data.

Findings from evidence-based forecasting: Methods for reducing forecast error

“…In the M3-Competition, RBF was run using automatic procedures (Adya et al 2001) and without any domain knowledge. RBF was the most accurate of the 22 methods for annual forecasts involving 645 series and six-year horizons (Makridakis and Hibon, 2000;Adya, et al 2000). Its symmetric MAPE (Mean Absolute Percentage Error) was 3.8% less than that for combining forecasts.…”

Replication and Extension of a Forecasting Decision Support System: An Empirical Examination of the Time Series Complexity Scoring Technique

“…The design of this Automated RBF (ARBF) set the stage for further replication and validation of RBF on a much larger data set of 3003 time series in Adya et al, (2000) as well as an extension that involved a simpler set of forecasting rules (Adya & Lusk, 2013a). The subject of our study, the CST developed by A&L, has benefited from this history of replications and extensions and presents the next enhancement to ARBF i.e., it systematically builds on the features developed and validated as part of the RBF and ARBF studies.…”

Development and validation of a rule-based time series complexity scoring technique to support design of adaptive forecasting DSS

“…Their study presented the Rule-Based Forecasting (RBF) system, an FDSS that relies on 18 features of time series to combine forecasts from four accepted forecasting methods: Random Walk (Naïve 1), OLS Linear Regression, Holt's two parameter exponential smoothing (ARIMA [0,0,2]), and Brown's exponential smoothing. These initial set of C&A features were validated and extended in studies such as, 43,44,45 thereby establishing strong theoretical and empirical foundation over two decades.…”

“…Second, the feature codings were validated in several extensions. 43,44,45 Third, forecast errors for RBF and Combining A provided a priori validated benchmarks for refinement and sensitivity analysis of the CST during development. The assumption that complex series will have lower forecast accuracy than simple ones formed the logical basis for calibrations and directional hypothesis formation and testing.…”

“…These rules were subjected to subsequent validation in several studies. 43,44,45 RBF rules, then, captured forecasting best practices in a robust knowledge base and were a fitting starting point for identifying CST rules. The following approach was applied to generate CST rules:…”