Last Layer Marginal Likelihood for Invariance Learning

Abstract

Data augmentation is often used to incorporate inductive biases into models. Traditionally, these are hand-crafted and tuned with cross validation. The Bayesian paradigm for model selection provides a path towards end-to-end learning of invariances using only the training data, by optimising the marginal likelihood. We work towards bringing this approach to neural networks by using an architecture with a Gaussian process in the last layer, a model for which the marginal likelihood can be computed. Experimentally, we improve performance by learning appropriate invariances in standard benchmarks, the low data regime and in a medical imaging task. Optimisation challenges for invariant Deep Kernel Gaussian processes are identified, and a systematic analysis is presented to arrive at a robust training scheme. We introduce a new lower bound to the marginal likelihood, which allows us to perform inference for a larger class of likelihood functions than before, thereby overcoming some of the training challenges that existed with previous approaches.

Relaxing Equivariance Constraints with Non-stationary Continuous Filters

“…In [4] and [8] symmetries are selected using a regularized training loss directly. Many symmetry discovery methods focus on learning invariances [27,4,22,26,10], which are easier to parameterize. This works offers a way to parameterize continuous equivariance constraints, which could allow extensions of symmetry discovery approaches to learnable equivariances.…”

“…Secondly, automatically learning symmetry structure from data is an interesting problem. Work in this field which often focuses on invariances [27,4,26,22,10], which are easier to parameterize than equivariances. Parameterizations that allow smooth and adjustable symmetry constraints could extends such methods to layer-by-layer learnable equivariances.…”