Statistical and Machine Learning forecasting methods: Concerns and ways forward

Abstract

Machine Learning (ML) methods have been proposed in the academic literature as alternatives to statistical ones for time series forecasting. Yet, scant evidence is available about their relative performance in terms of accuracy and computational requirements. The purpose of this paper is to evaluate such performance across multiple forecasting horizons using a large subset of 1045 monthly time series used in the M3 Competition. After comparing the post-sample accuracy of popular ML methods with that of eight traditional statistical ones, we found that the former are dominated across both accuracy measures used and for all forecasting horizons examined. Moreover, we observed that their computational requirements are considerably greater than those of statistical methods. The paper discusses the results, explains why the accuracy of ML models is below that of statistical ones and proposes some possible ways forward. The empirical results found in our research stress the need for objective and unbiased ways to test the performance of forecasting methods that can be achieved through sizable and open competitions allowing meaningful comparisons and definite conclusions.

Scalable high-resolution forecasting of sparse spatiotemporal events with kernel methods: A winning solution to the NIJ “Real-Time Crime Forecasting Challenge”

“…Supervised learning methods are widely used within non-spatiotemporal applications. However, they are less commonly used within the applied spatial (Heaton et al, 2018), time series (Makridakis, Spiliotis and Assimakopoulos, 2018), and crime forecasting domains. In crime forecasting, KDE-based forecasting approaches remain the most common forecasting techniques used (Gorr, Olligschlaeger and Thompson, 2003;Gorr, 2009;Chainey, Tompson and Uhlig, 2008a;Caplan, Kennedy and Miller, 2011;Berk et al, 2018).…”

Bandwidth Prediction Schemes for Defining Bitrate Levels in SDN-enabled Adaptive Streaming

“…In the proposed architecture, we implemented the Network Agent, that carries instructions or defined policies from the QoE management plane and translate those policies into a set of rules or actions on the data forwarding plane. Further, the Restful API [15] has been employed to provide communication between the QoE management plane and the control plane, while OpenFlow acts as the southbound interface protocol.…”

Anomaly Detection with HMM Gauge Likelihood Analysis

“…This has been done several times, for recent results see e.g. (Makridakis et al, 2018b), the NN3 competition (Crone et al, 2005), and the M4 competition (Makridakis et al, 2018a). In general, deep learning seems like a promising direction for anomaly detection in time series, especially if long term correlations between events are present.…”