On Comparing Cross-Validated Forecasting Models with a Novel Fuzzy-TOPSIS Metric: A COVID-19 Case Study

Abstract

Time series cross-validation is a technique to select forecasting models. Despite the sophistication of cross-validation over single test/training splits, traditional and independent metrics, such as Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), are commonly used to assess the model’s accuracy. However, what if decision-makers have different models fitting expectations to each moment of a time series? What if the precision of the forecasted values is also important? This is the case of predicting COVID-19 in Amapá, a Brazilian state in the Amazon rainforest. Due to the lack of hospital capacities, a model that promptly and precisely responds to notable ups and downs in the number of cases may be more desired than average models that only have good performances in more frequent and calm circumstances. In line with this, this paper proposes a hybridization of the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and fuzzy sets to create a similarity metric, the closeness coefficient (CC), that enables relative comparisons of forecasting models under heterogeneous fitting expectations and also considers volatility in the predictions. We present a case study using three parametric and three machine learning models commonly used to forecast COVID-19 numbers. The results indicate that the introduced fuzzy similarity metric is a more informative performance assessment metric, especially when using time series cross-validation.

COVID-19 ICU demand forecasting: A two-stage Prophet-LSTM approach

“…A good explanation of those models and their main mathematical formulations are presented by Souza et al. [56] , Kadri and Abdennbi [57] , and Sezer et al. [49] .…”

Traffic Flow Prediction Method Based on Seasonal Characteristics and SARIMA-NAR Model

“…In addition, due to the impact of the COVID-19 epidemic, controlling of urban traffic has been upgraded, and people's travel has been restricted. When a city enters epidemic prevention and control from a state of congestion, it needs a timely updated traffic control strategy to reduce the impact on society and the economy [1,2]. Predicting and monitoring the traffic flow during the epidemic can effectively improve the control level of the urban transportation network and examine the impact of the traffic situation on the spread of the epidemic.…”

Integrating FSE and AHP to Identify Valuable Customer Needs by Service Quality Analysis

“…FSE is a widely used method in fuzzy mathematics; the MCDM approach is based on fuzzy mathematics and centered on the grade of the membership function of fuzzy mathematics [14,15]. FSE transfers the qualitative evaluation into the quantitative evaluation to offer a synthetic evaluation and then implements an evaluation of a single index.…”