Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches

Abstract

Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies. Unfortunately, due to the large number of weights, all the examples in a mini-batch typically share the same weight perturbation, thereby limiting the variance reduction effect of large mini-batches. We introduce flipout, an efficient method for decorrelating the gradients within a mini-batch by implicitly sampling pseudo-independent weight perturbations for each example. Empirically, flipout achieves the ideal linear variance reduction for fully connected networks, convolutional networks, and RNNs. We find significant speedups in training neural networks with multiplicative Gaussian perturbations. We show that flipout is effective at regularizing LSTMs, and outperforms previous methods. Flipout also enables us to vectorize evolution strategies: in our experiments, a single GPU with flipout can handle the same throughput as at least 40 CPU cores using existing methods, equivalent to a factor-of-4 cost reduction on Amazon Web Services.

Exploring the Limits of Epistemic Uncertainty Quantification in Low-Shot Settings

“…Uncertainty Methods. We evaluate MC-Dropout (DO) [4], MC-DropConnect (DC) [11], Deep Ensembles (DE) [10], Direct Uncertainty Quantification (DUQ) [15], Variational Inference with Flipout (VI) [16], and Gradient-based uncertainty (GD) [12]. This selection covers scalable as well as approximate methods and recent advances.…”

“…This transforms the model into a stochastic one. Flipout [16] is used as an additional formulation on top of a stochastic perturbation model that reduces variance, greatly improving learning stability and performance.…”

“…Bayesian Deep Learning promises good uncertainty estimates [17], but methods often rely in approximations to the bayesian posterior, or quantify uncertainty in approximate [16] [1] or non-bayesian ways [10]. Evaluating the quality of output uncertainty is difficult as there are no labels.…”

Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning

“…Comparing with the classical strategy that perturbs weight in the entire space [40,16,42,10], we focus on characterizing the weight loss landscape on the new task with respect to the important subspace representing old task. The important subspace can be effectively calculated based on the examples sampled from replay buffer M after each task training.…”

Uncertainty Quantification in Neural Differential Equations

“…1. We propose an adaptation of the four state-of-the-art UQ methods in deep learning -Bayes By Backprop [3], Flipout [24], Neural Linear Model [23,15], and Deep Evidential Regression [2,13] -to the case of solving DEs. A DE can be expressed as Lu − f = 0, where L is the differential operator, u(x) is the solution that we wish to find on some (possibly multidimensional) domain x, and f is a known forcing function.…”