Sparse MoEs meet Efficient Ensembles

Abstract

Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, lead to strong performance. We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that the two approaches have complementary features whose combination is beneficial. Then, we present partitioned batch ensembles, an efficient ensemble of sparse MoEs that takes the best of both classes of models. Extensive experiments on fine-tuned vision Transformers demonstrate the accuracy, log-likelihood, few-shot learning, robustness, and uncertainty improvements of our approach over several challenging baselines. Partitioned batch ensembles not only scale to models with up to 2.7B parameters, but also provide larger performance gains for larger models. * Work done as a Google Brain intern. † Work done while at Google Brain.

SKDBERT: Compressing BERT via Stochastic Knowledge Distillation

“…We perform extensive ablation experiments to show the effectiveness of SKDBERT in terms of teacher ensemble, sampling distribution, KD paradigm, extra learning procedure and distillation objective. Appropriately increasing the number of teachers can effectively improve the diversity of prediction (Allingham et al 2021) for obtaining better performance. As a result, we discuss the effectiveness of weak teachers (e.g., T 01 to T 03 for SKDBERT 4 , T 01 to T 06 for SKDBERT 6 ).…”

LiDAR Dataset Distillation within Bayesian Active Learning Framework Understanding the Effect of Data Augmentation

“…Key future challenges Quantifying data redundancy shall be investigated in our future study based on work by authors in (Birodkar et al, 2019), (Guo et al, 2021). To improve the heuristic function, recent work (Lakshminarayanan et al, 2016) (Allingham et al, 2021 on explicit ensembles shows strong results for uncertainty computing, and (Aghdam et al, 2019) show that adding temporal reasoning can be beneficial for data selection on object detection task. We aim to further our study by experiment on different budget sizes, while testing on the complete Semantic-KITTI dataset.…”