Hyperparameter Estimation in Bayesian MAP Estimation: Parameterizations and Consistency

Abstract

The Bayesian formulation of inverse problems is attractive for three primary reasons: it provides a clear modelling framework; means for uncertainty quantification; and it allows for principled learning of hyperparameters. The posterior distribution may be explored by sampling methods, but for many problems it is computationally infeasible to do so. In this situation maximum a posteriori (MAP) estimators are often sought. Whilst these are relatively cheap to compute, and have an attractive variational formulation, a key drawback is their lack of invariance under change of parameterization. This is a particularly significant issue when hierarchical priors are employed to learn hyperparameters. In this paper we study the effect of the choice of parameterization on MAP estimators when a conditionally Gaussian hierarchical prior distribution is employed. Specifically we consider the centred parameterization, the natural parameterization in which the unknown state is solved for directly, and the noncentred parameterization, which works with a whitened Gaussian as the unknown state variable, and arises when considering dimension-robust MCMC algorithms; MAP estimation is well-defined in the nonparametric setting only for the noncentred parameterization. However, we show that MAP estimates based on the noncentred parameterization are not consistent as estimators of hyperparameters; conversely, we show that limits of finite-dimensional centred MAP estimators are consistent as the dimension tends to infinity. We also consider empirical Bayesian hyperparameter estimation, show consistency of these estimates, and demonstrate that they are more robust with respect to noise than centred MAP estimates. An underpinning concept throughout is that hyperparameters may only be recovered up to measure equivalence, a well-known phenomenon in the context of the Ornstein-Uhlenbeck process.

Convergence rates for ansatz‐free data‐driven inference in physically constrained problems

“…Finally, we choose an annealing schedule 𝛽 ℎ such that 𝛽 ℎ 𝜖 2 ℎ → 0, ensuring that Assumption (iii) holds. Specifically, we may choose 𝛽 ℎ ∼ 𝜖 −1 ℎ , as suggested by (11) to obtain an approximation with the claimed quantitative convergence rate.…”

Convergence rates for ansatz-free data-driven inference in physically constrained problems

“…They also provide a natural means of approximating general measures in the sense of weak convergence. Specifically, we consider a sequence of material likelihood measures µ D,h of the form given in (5). We begin by showing that, if (µ β h ) is uniformly bounded and tight and (µ D,h ) suitably approximates µ D , then (µ h,β h ) is also uniformly bounded and tight.…”

“…We begin by showing that, if (µ β h ) is uniformly bounded and tight and (µ D,h ) suitably approximates µ D , then (µ h,β h ) is also uniformly bounded and tight. (5). Let β h → +∞ and h → 0.…”

“…A common choice for g D,h are neural networks, in the context of machine learning [7], whereas a common choice of f D,h is Gaussian [10,4]. Common methods of regression used to determine the parameters from the data include classical methods of statistical inference such as maximum likelihood [9,5], variational approaches based on the introduction of a loss function [14], measure-theoretical approaches based, e. g., on the Wasserstein distance [1] or the Kullback-Leibler discrepancy [12].…”

Hybrid iterative ensemble smoother for history matching of hierarchical models

“…In their experiments, which included nonstationary hyperparameters, they concluded that using a hierarchical approach with a non-centered parameterization was significantly better than a hierarchical approach with a centered parameterization and bothh hierarchical approaches were better than a non-hierarchical approach. Subsequently, Dunlop et al Dunlop et al (2020) showed that for MAP estimation of the hyperparameters in a linear inverse problem, the centered parameterization is to be preferred when the goal is MAP estimation as opposed to uncertainty quantification.…”