Taming the Long Tail of Deep Probabilistic Forecasting

Abstract

Deep probabilistic forecasting is gaining attention in numerous applications ranging from weather prognosis, through electricity consumption estimation, to autonomous vehicle trajectory prediction. However, existing approaches focus on improvements on the most common scenarios without addressing the performance on rare and difficult cases. In this work, we identify a long tail behavior in the performance of state-of-the-art deep learning methods on probabilistic forecasting. We present two moment-based tailedness measurement concepts to improve performance on the difficult tail examples: Pareto Loss and Kurtosis Loss. Kurtosis loss is a symmetric measurement as the fourth moment about the mean of the loss distribution. Pareto loss is asymmetric measuring right tailedness, modeling the loss using a generalized Pareto distribution (GPD). We demonstrate the performance of our approach on several real-world datasets including time series and spatiotemporal trajectories, achieving significant improvements on the tail examples.

Prediction of Social Dynamic Agents and Long-Tailed Learning Challenges: A Survey

“…Despite the difficulty in creating and quantifying a dataset that has a long tail across all relevant measures, the creation of such a dataset would have many benefits for evaluating and comparing separate long-tailed learning techniques. For example, since there is no one scale across which rareness can be measured, many algorithms (e.g., Makansi et al, 2021;Kozerawski et al, 2022) define and optimize for their own scale, making the results difficult to compare. However, with a dataset that's been curated to have a long tail across many different scales, methods optimizing for different scales can be compared.…”

“…Within motion prediction, Makansi et al (2021) and Kozerawski et al (2022) directly address long-tailed learning in trajectory prediction, while Li et al (2021b) simply show that injecting logic rules by adding cross-walks, traffic lights, and left/right turn only lanes into the map and making them hard rules instead of suggestions to be used as input, reduces the long tail of the error distribution, as shown in Figure 3 of Li et al (2021b). Although Anderson et al (2019) don't directly address dataset imbalance, they develop a data augmentation method that could be used to upsample uncommon trajectories by gen-erating trajectories from dataset statistics and adding random transformations to increase the variety and number of trajectories.…”

“…In Makansi et al (2021), classes are defined by how easy it is to predict the future trajectory through a physics-based Kalman filter: rare and important trajectories are assumed to be the ones which are difficult to predict using simple kinematics. Kozerawski et al (2022), on the other hand, compare two novel loss terms that up-weight rare, high error examples: a regularization term which improves performance slightly in average and rare cases, and a kurtosis term which significantly improves only the worst error. The regularization term includes hyperparameters that assume a fixed shape for the error distribution while the kurtosis term uses batch statistics to estimate the error distribution.…”

“…The regularization term includes hyperparameters that assume a fixed shape for the error distribution while the kurtosis term uses batch statistics to estimate the error distribution. While Makansi et al (2021) and Kozerawski et al (2022) show small improvements in averaged metrics, it is difficult to compare improvements in the long tail as each paper defines and optimizes for their own scale of uncommonness. Therefore, a standardized method of evaluating long tailed performance must be defined for trajectory prediction.…”

“…Most long-tailed learning algorithms focus on dataset imbalance across the distribution of labels or performances. Classification techniques separate examples based on how common the class label is, while some trajectory prediction algorithms separate trajectories by how different their error is from that of other examples (e.g., Makansi et al, 2021;Kozerawski et al, 2022). However, one underexplored aspect of long-tailedness is the influence of environmental factors.…”