The Deep Weight Prior

Abstract

Bayesian inference is known to provide a general framework for incorporating prior knowledge or specific properties into machine learning models via carefully choosing a prior distribution. In this work, we propose a new type of prior distributions for convolutional neural networks, deep weight prior (dwp), that exploit generative models to encourage a specific structure of trained convolutional filters e.g., spatial correlations. We define dwp in a form of an implicit distribution and propose a method for variational inference with such type of implicit priors. In experiments, we show that dwp improves the performance of Bayesian neural networks when training data are limited, and initialization of weights with samples from dwp accelerates training of conventional convolutional neural networks.

Bayesian Neural Network Priors Revisited

“…However, most modern applications of BNNs still relied on simple Gaussian priors. Although a few different priors have been proposed for BNNs, these were mostly designed for specific tasks (Atanov et al, 2018;Ghosh & Doshi-Velez, 2017;Overweg et al, 2019;Nalisnick, 2018;Cui et al, 2020;Hafner et al, 2020) or relied heavily on non-standard inference methods (Sun et al, 2019;Ma et al, 2019;Karaletsos & Bui, 2020;Pearce et al, 2020). Moreover, while many interesting distributions have been proposed as variational posteriors for BNNs (Louizos & Welling, 2017;Swiatkowski et al, 2020;Dusenberry et al, 2020;Aitchison et al, 2020), these approaches have still used Gaussian priors.…”

“…BNN priors. Finally, previous work has investigated the performance implications of neural network priors chosen without reference to the empirical distributions of SGD-trained networks (Ghosh & Doshi-Velez, 2017;Wu et al, 2018;Atanov et al, 2018;Nalisnick, 2018;Overweg et al, 2019;Farquhar et al, 2019;Cui et al, 2020;Rothfuss et al, 2020;Hafner et al, 2020;Matsubara et al, 2020;Tran et al, 2020;Garriga-Alonso & van der Wilk, 2021). While these priors might in certain circumstances offer performance improvements, they did not offer a recipe for finding potentially valuable features to incorporate into the weight priors.…”

A review of uncertainty quantification in deep learning: Techniques, applications and challenges

“…Bayesian inference facilitates a general framework for incorporating specific properties or prior knowledge into machine learning techniques through selecting a prior distribution carefully. Atanov et al [387] presented a novel type of prior distributions for CNN, deep weight prior (DWP), that examined generative models to persuade a certain structure of trained convolutional filters. They devised a technique for VI with implicit priors and denoted DWP in a form of an implicit distribution.…”

Statistical physics of unsupervised learning with prior knowledge in neural networks

“…The Bayesian brain hypothesis [4] was widely used to model sensorimotor behavior [5,6], perceptual decision making [7], object perception in visual cortex [8,9], and even cognition [10]. Bayesian inference is also a principled computational framework in deeplearning-based machine learning [11][12][13]. When the prior beliefs are taken into account, the unsupervised learning meets Bayesian inference.…”

“…Thus incorporating prior beliefs from past experiences into unsupervised learning is a common and fundamental characteristic of the computation in the brain or artificial neural networks. However, current studies mostly focused on neural implementations of the Bayesian inference [4,14], or focused on designing scalable Bayesian learning algorithms for deep networks [12,13], making a scientific understanding of unsupervised learning with prior knowledge lag far behind its neural implementations or engineering applications.…”