Deep Ensembles Work, But Are They Necessary?

Abe, Taiga; Buchanan, E. Kelly; Pleiss, Geoff; Zemel, Richard S.; Cunningham, John P.

doi:10.48550/arxiv.2202.06985

Cited by 5 publications

(10 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, generating an ensemble with a size of a least 10 appears to be a sensible choice, with only minor improvements being observed for more than 20 members. This corresponds to the results in Fort et al (2019) and ensemble sizes typically chosen in the literature (Lakshminarayanan et al, 2017;Rasp and Lerch, 2018), but the benefits of generating more ensemble members need to be balanced against the computational costs, and sometimes smaller ensembles have been suggested (Ovadia et al, 2019;Abe et al, 2022). Third, aggregating forecast distributions via VI is often superior to the LP.…”

Section: Discussionsupporting

confidence: 54%

“…Technically, we here use the unified PIT, a generalization proposed inVogel et al (2018), due to the format of some of the aggregated forecast distributions.2 For example,Lichtendahl et al (2013) andAbe et al (2022) show that the score of the LP forecast is at least as good as the average score of the individual components in terms of different proper scoring rules.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Aggregating distribution forecasts from deep ensembles

Schulz¹,

Lerch²

2022

Preprint

View full text Add to dashboard Cite

The importance of accurately quantifying forecast uncertainty has motivated much recent research on probabilistic forecasting. In particular, a variety of deep learning approaches has been proposed, with forecast distributions obtained as output of neural networks. These neural network-based methods are often used in the form of an ensemble based on multiple model runs from different random initializations, resulting in a collection of forecast distributions that need to be aggregated into a final probabilistic prediction. With the aim of consolidating findings from the machine learning literature on ensemble methods and the statistical literature on forecast combination, we address the question of how to aggregate distribution forecasts based on such 'deep ensembles'. Using theoretical arguments, simulation experiments and a case study on wind gust forecasting, we systematically compare probability-and quantile-based aggregation methods for three neural network-based approaches with different forecast distribution types as output. Our results show that combining forecast distributions can substantially improve the predictive performance. We propose a general quantile aggregation framework for deep ensembles that shows superior performance compared to a linear combination of the forecast densities. Finally, we investigate the effects of the ensemble size and derive recommendations of aggregating distribution forecasts from deep ensembles in practice.

show abstract

Section: Discussionsupporting

confidence: 54%

mentioning

confidence: 99%

Aggregating distribution forecasts from deep ensembles

Schulz¹,

Lerch²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Specifically, we propose to train a set of GNNs {GNN 1 , GNN 2 , ..., GNN }. Given a set of training nodes D ⊆ V 1 , we generate bootstraps {D (1) , D (2) , ..., D ( ) } subject to the constraint that |D (i) Ω k u | = 1 for all u ∈ D and i ≤ (i.e., each bootstrap contains exactly one relative of each training node). This constraint allows us to avoid the overrepresentation problem, address CH1 by sampling with replacement, and address CH2 by training each GNN i on different neighborhood subspaces as represented in D (i) .…”

Section: Why Deep Graph Ensembles?mentioning

confidence: 99%

“…DGE-batch* therefore represents a traditional GNN that is trained with awareness of CH1 and CH3, and addresses CH2 in a manner similar to a READOUT function [44]. We considered DGE-batch* an important variant to evaluate because its performance illuminates the importance of using an ensemble instead of a single, more complex model to solve CH2 [1].…”

Section: Training and Inferencementioning

confidence: 99%

Deep Ensembles for Graphs with Higher-order Dependencies

Krieg¹,

Burgis²,

Soga³

et al. 2022

Preprint

View full text Add to dashboard Cite

Graph neural networks (GNNs) continue to achieve state-of-the-art performance on many graph learning tasks, but rely on the assumption that a given graph is a sufficient approximation of the true neighborhood structure. In the presence of higher-order sequential dependencies, we show that the tendency of traditional graph representations to underfit each node's neighborhood causes existing GNNs to generalize poorly. To address this, we propose a novel Deep Graph Ensemble (DGE), which captures neighborhood variance by training an ensemble of GNNs on different neighborhood subspaces of the same node within a higher-order network structure. We show that DGE consistently outperforms existing GNNs on semisupervised and supervised tasks on four real-world data sets with known higher-order dependencies, even under a similar parameter budget. We demonstrate that learning diverse and accurate base classifiers is central to DGE's success, and discuss the implications of these findings for future work on GNNs.

show abstract

“…We showcase these properties theoretically and demonstrate the benefits of transformation ensembles empirically on several semi-structured data sets. With transformation ensembles we are able to provide empirical evidence for answering open questions in deep ensembling (Abe et al, 2022). For instance, the increased flexibility of classical deep ensembles over their members does not seem to be necessary for improving prediction performance or allowing uncertainty quantification.…”

Section: Our Contributionmentioning

confidence: 99%

Deep interpretable ensembles

Kook¹,

Götschi²,

Baumann³

et al. 2022

Preprint

View full text Add to dashboard Cite

Ensembles improve prediction performance and allow uncertainty quantification by aggregating predictions from multiple models. In deep ensembling, the individual models are usually black box neural networks, or recently, partially interpretable semi-structured deep transformation models. However, interpretability of the ensemble members is generally lost upon aggregation. This is a crucial drawback of deep ensembles in high-stake decision fields, in which interpretable models are desired. We propose a novel transformation ensemble which aggregates probabilistic predictions with the guarantee to preserve interpretability and yield uniformly better predictions than the ensemble members on average. Transformation ensembles are tailored towards interpretable deep transformation models but are applicable to a wider range of probabilistic neural networks. In experiments on several publicly available data sets, we demonstrate that transformation ensembles perform on par with classical deep ensembles in terms of prediction performance, discrimination, and calibration. In addition, we demonstrate how transformation ensembles quantify both aleatoric and epistemic uncertainty, and produce minimax optimal predictions under certain conditions.

show abstract

Deep Ensembles Work, But Are They Necessary?

Cited by 5 publications

References 25 publications

Aggregating distribution forecasts from deep ensembles

Aggregating distribution forecasts from deep ensembles

Deep Ensembles for Graphs with Higher-order Dependencies

Deep interpretable ensembles

Contact Info

Product

Resources

About