The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks

Abstract

Bayesian neural networks attempt to combine the strong predictive performance of neural networks with formal quantification of uncertainty associated with the predictive output in the Bayesian framework. However, it remains unclear how to endow the parameters of the network with a prior distribution that is meaningful when lifted into the output space of the network. A possible solution is proposed that enables the user to posit an appropriate covariance function for the task at hand. Our approach constructs a prior distribution for the parameters of the network, called a ridgelet prior, that approximates the posited covariance structure in the output space of the network. The approach is rooted in the ridgelet transform and we establish both finite-sample-size error bounds and the consistency of the approximation of the covariance function in a limit where the number of hidden units is increased. Our experimental assessment is limited to a proof-of-concept, where we demonstrate that the ridgelet prior can out-perform an unstructured prior on regression problems for which an informative covariance function can be a priori provided.

Bayesian Neural Network Priors Revisited

“…BNN priors. Finally, previous work has investigated the performance implications of neural network priors chosen without reference to the empirical distributions of SGD-trained networks (Ghosh & Doshi-Velez, 2017;Wu et al, 2018;Atanov et al, 2018;Nalisnick, 2018;Overweg et al, 2019;Farquhar et al, 2019;Cui et al, 2020;Rothfuss et al, 2020;Hafner et al, 2020;Matsubara et al, 2020;Tran et al, 2020;Garriga-Alonso & van der Wilk, 2021). While these priors might in certain circumstances offer performance improvements, they did not offer a recipe for finding potentially valuable features to incorporate into the weight priors.…”

Priors in Bayesian Deep Learning: A Review

“…If one wants to forego the need for a well-defined divergence, one can also use a hypernetwork (Ha et al, 2016;Krueger et al, 2017) as an implicit distribution of BNN weights and then train the network to match the GP samples on a certain set of function outputs (Flam-Shepherd et al, 2018). Finally, it has recently been discovered that the ridgelet transform (Candes, 1998) can be used to approximate GP function-space distributions with BNN weight-space distributions (Matsubara et al, 2020). As a sidenote, it should be noted that the reverse can actually be achieved more easily, namely, fitting a GP to the outputs of a BNN , which can also be of interest in certain applications.…”

Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel

“…spatial distance) as GPs do not scale well with the number of samples for high-dimensional inputs [4,5]. To overcome this issue, Pearce et al [32] and Matsubara et al [27] developed BNN analogue for composite GPs. Similar to these studies, our ICK framework can also be viewed as a simulation for composite GPs.…”

“…Related to our proposed methodology, Pearce et al [32] exploited the fact that a Bayesian neural network (BNN) approximates a GP to construct additive and multiplicative kernels, but they were limited to specific predefined kernels. Matsubara et al [27] then resolved this limitation by constructing priors of BNN parameters based on the ridgelet transform and its dual, but they did not explicitly show how their approach works for data with multiple sources of information. To our knowledge, none of these existing approaches allows a modeler to choose any appropriate kernel of known prior information from multiple sources.…”