High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes

Salinas, David; Bohlke-Schneider, Michael; Callot, Laurent; Medico, Roberto; Gasthaus, Jan

doi:10.48550/arxiv.1910.03002

Cited by 5 publications

(27 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address the limitation of ES in multidimensional data, Salinas et al [11] introduced CRPS-Sum for evaluating a multivariate probabilistic forecasting model. CRPS-Sum is a proper scoring rule, and it is not a strictly proper.…”

Section: Crps-summentioning

confidence: 99%

“…The CRPS-Sum has been wide welcomed by the scientific community and many researches have used it to report the performance of their models [11,12,14,13]. However, the capabilities of CRPS-Sum have not been investigated thoroughly unlike the vast studied dedicated to the properties of ES and CRPS [18,19,22].…”

Section: Investigating Crps-sum Propertiesmentioning

confidence: 99%

“…To calculate CRPS-Sum, first we sum the time-series over the dimensions [11]. Although this aggregation let us turn a multivariate time-series into a univariate one, we lose important information concerning the performance of the model on each dimension.…”

Section: The Effect Of Summation In Crps-summentioning

confidence: 99%

“…Due to the recent upsurge in Deep Learning, several approaches have been proposed which are based or inspired by these powerful neural network. To name some of these techniques, there are DeepAR [10], Low-Rank Gaussian Copula Processes [11], Conditioned Normalizing Flows [12], Normalizing Kalman Filter [13], Denoising Diffusion Model [14], and Conditional Generative Adversarial Networks [15,16]. In many proposed solutions, we do not have direct access to the probability distribution of the model over future values.…”

Section: Introductionmentioning

confidence: 99%

“…Recently, Salinas et al [11] suggest CRPS-Sum as a new proper multivariate scoring rule. This scoring rule has been well-received in the scientific community [11,12,14,13]. The properties of CRPS-Sum are not studied so far.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability

Koochali¹,

Schichtel²,

Dengel³

et al. 2022

Preprint

View full text Add to dashboard Cite

The recent developments in the machine learning domain have enabled the development of complex multivariate probabilistic forecasting models. Therefore, it is pivotal to have a precise evaluation method to gauge the performance and predictability power of these complex methods. To do so, several evaluation metrics have been proposed in the past (such as Energy Score, Dawid-Sebastiani score, variogram score), however, they cannot reliably measure the performance of a probabilistic forecaster. Recently, CRPS-sum has gained a lot of prominence as a reliable metric for multivariate probabilistic forecasting. This paper presents a systematic evaluation of CRPS-sum to understand its discrimination ability. We show that the statistical properties of target data affect the discrimination ability of CRPS-Sum. Furthermore, we highlight that CRPS-Sum calculation overlooks the performance of the model on each dimension. These flaws can lead us to an incorrect assessment of model performance. Finally, with experiments on the real-world dataset, we demonstrate that the shortcomings of CRPS-Sum provide a misleading indication of the probabilistic forecasting performance method. We show that it is easily possible to have a better CRPS-Sum for a dummy model, which looks like random noise, in comparison to the state-of-theart method.

show abstract

Section: Crps-summentioning

confidence: 99%

Section: Investigating Crps-sum Propertiesmentioning

confidence: 99%

Section: The Effect Of Summation In Crps-summentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability

Koochali¹,

Schichtel²,

Dengel³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

SLOTH: Structured Learning and Task-Based Optimization for Time Series Forecasting on Hierarchies

Zhou,

Pan,

et al. 2023

AAAI

View full text Add to dashboard Cite

Multivariate time series forecasting with hierarchical structure is widely used in real-world applications, e.g., sales predictions for the geographical hierarchy formed by cities, states, and countries. The hierarchical time series (HTS) forecasting includes two sub-tasks, i.e., forecasting and reconciliation. In the previous works, hierarchical information is only integrated in the reconciliation step to maintain coherency, but not in forecasting step for accuracy improvement. In this paper, we propose two novel tree-based feature integration mechanisms, i.e., top-down convolution and bottom-up attention to leverage the information of the hierarchical structure to improve the forecasting performance. Moreover, unlike most previous reconciliation methods which either rely on strong assumptions or focus on coherent constraints only, we utilize deep neural optimization networks, which not only achieve coherency without any assumptions, but also allow more flexible and realistic constraints to achieve task-based targets, e.g., lower under-estimation penalty and meaningful decision-making loss to facilitate the subsequent downstream tasks. Experiments on real-world datasets demonstrate that our tree-based feature integration mechanism achieves superior performances on hierarchical forecasting tasks compared to the state-of-the-art methods, and our neural optimization networks can be applied to real-world tasks effectively without any additional effort under coherence and task-based constraints.

show abstract

Random Noise vs. State-of-the-Art Probabilistic Forecasting Methods: A Case Study on CRPS-Sum Discrimination Ability

et al. 2022

View full text Add to dashboard Cite

The recent developments in the machine-learning domain have enabled the development of complex multivariate probabilistic forecasting models. To evaluate the predictive power of these complex methods, it is pivotal to have a precise evaluation method to gauge the performance and predictability power of these complex methods. To do so, several evaluation metrics have been proposed in the past (such as the energy score, Dawid–Sebastiani score, and variogram score); however, these cannot reliably measure the performance of a probabilistic forecaster. Recently, CRPS-Sum has gained a lot of prominence as a reliable metric for multivariate probabilistic forecasting. This paper presents a systematic evaluation of CRPS-Sum to understand its discrimination ability. We show that the statistical properties of target data affect the discrimination ability of CRPS-Sum. Furthermore, we highlight that CRPS-Sum calculation overlooks the performance of the model on each dimension. These flaws can lead us to an incorrect assessment of model performance. Finally, with experiments on real-world datasets, we demonstrate that the shortcomings of CRPS-Sum provide a misleading indication of the probabilistic forecasting performance method. We illustrate that it is easily possible to have a better CRPS-Sum for a dummy model, which looks like random noise, in comparison to the state-of-the-art method.

show abstract

High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes

Cited by 5 publications

References 7 publications

Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability

Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability

SLOTH: Structured Learning and Task-Based Optimization for Time Series Forecasting on Hierarchies

Random Noise vs. State-of-the-Art Probabilistic Forecasting Methods: A Case Study on CRPS-Sum Discrimination Ability

Contact Info

Product

Resources

About