Optimal Low-rank Approximations of Bayesian Linear Inverse Problems

Abstract

Abstract. In the Bayesian approach to inverse problems, data are often informative, relative to the prior, only on a low-dimensional subspace of the parameter space. Significant computational savings can be achieved by using this subspace to characterize and approximate the posterior distribution of the parameters. We first investigate approximation of the posterior covariance matrix as a low-rank update of the prior covariance matrix. We prove optimality of a particular update, based on the leading eigendirections of the matrix pencil defined by the Hessian of the negative log-likelihood and the prior precision, for a broad class of loss functions. This class includes the Förstner metric for symmetric positive definite matrices, as well as the Kullback-Leibler divergence and the Hellinger distance between the associated distributions. We also propose two fast approximations of the posterior mean and prove their optimality with respect to a weighted Bayes risk under squared-error loss. These approximations are deployed in an offline-online manner, where a more costly but data-independent offline calculation is followed by fast online evaluations. As a result, these approximations are particularly useful when repeated posterior mean evaluations are required for multiple data sets. We demonstrate our theoretical results with several numerical examples, including high-dimensional X-ray tomography and an inverse heat conduction problem. In both of these examples, the intrinsic low-dimensional structure of the inference problem can be exploited while producing results that are essentially indistinguishable from solutions computed in the full space.Key words. inverse problems, Bayesian inference, low-rank approximation, covariance approximation, Förstner-Moonen metric, posterior mean approximation, Bayes risk, optimality 1. Introduction. In the Bayesian approach to inverse problems, the parameters of interest are treated as random variables, endowed with a prior probability distribution that encodes information available before any data are observed. Observations are modeled by their joint probability distribution conditioned on the parameters of interest, which defines the likelihood function and incorporates the forward model and a stochastic description of measurement or model errors. The prior and likelihood then combine to yield a probability distribution for the parameters conditioned on the observations, i.e., the posterior distribution. While this formulation is quite general, essential features of inverse problems bring additional structure to the Bayesian update. The prior distribution often encodes some kind of smoothness or correlation among the inversion parameters; observations typically are finite, few in number, and corrupted by noise; and the observations are indirect, related to the inversion parameters by the action of a forward operator that destroys some information. A key consequence of these features is that the data may be informative, relative to the prior, only on a low-dimensional subspace o...

Top-down constraints on global N<sub>2</sub>O emissions at optimal resolution: application of a new dimension reduction technique

“…The method, based on synthesis and advancement of recent work in this area (Flath et al, 2011;Bui-Thanh et al, 2012;Spantini et al, 2015) by Bousserez and Henze (2017), uses an optimal low-rank projection of the inverse problem that maximizes the observational constraints. Specifically, for a given dimension k, the optimal reduced space (Spantini et al, 2015;Bousserez and Henze, 2017) is spanned by the first k eigenvectors of the prior-preconditioned Hessian G (Flath et al, 2011):…”

“…The trace of the averaging kernel gives the total degrees of freedom, i.e., the number of independent pieces of information that can be obtained in the inversion framework. The posterior mean estimate of x can also be directly calculated from analytical formulas using the eigenvectors of G (Spantini et al, 2015;Bouserez and Henze, 2017). However, to impose a positivity constraint on the emissions, we rely here on the variational minimization framework as in the standard 4D-Var case.…”

Sampling via Measure Transport: An Introduction

“…A further source of low dimensionality in transports is low-rank structure, i.e., situations where a map departs from the identity only on a low-dimensional subspace of the input space [78]. This situation is fairly common in large-scale Bayesian inverse problems where the data are informative, relative to the prior, only about a handful of directions in the parameter space [25,79].…”

Model reduction from partial observations

“…In Section 4.3, we will emphasize that such a choice for M prior allows for an easy implementation of the sampling of M prior ∩ H h . In order to show that (15) can (for example) be obtained from standard model-order reduction techniques, let us first consider the case where the uncertainty one has on the set of possible solutions of the PPDE (i.e., M) is due to an imperfect knowledge of the set of feasible parameters Θ (The general case will be discussed at the end of this section). Although Θ may not be precisely known, an information the practitioner usually has at its disposal is that Θ is contained in some larger set Θ relax , i.e., Θ ⊆ Θ relax .…”

“…This approach was recently used in the literature. [11][12][13][14][15] Maday et al suggested to iteratively enrich the approximation subspace by using a posteriori estimates of some elements of  (the term "a posteriori" refers here to the fact that the estimates stem from the combination of partial observations and some prior knowledge on ). 11 In a previous work, 12 the authors of the present work refined this approach in a Bayesian framework: they proposed to include the uncertainty inherent to the a posteriori estimates in the reduction process.…”