Revisiting Maximum-A-Posteriori Estimation in Log-Concave Models

Abstract

Maximum-a-posteriori (MAP) estimation is the main Bayesian estimation methodology in imaging sciences, where high dimensionality is often addressed by using Bayesian models that are log-concave and whose posterior mode can be computed efficiently by convex optimisation. However, despite its success and wide adoption, MAP estimation is not theoretically well understood yet. In particular, the prevalent view in the community is that MAP estimation is not proper Bayesian estimation in the sense of Bayesian decision theory because it does not minimise a meaningful expected loss function (unlike the minimum mean squared error (MMSE) estimator that minimises the mean squared loss). This paper addresses this theoretical gap by presenting a general decision-theoretic derivation of MAP estimation in Bayesian models that are log-concave. A main novelty is that our analysis is based on differential geometry, and proceeds as follows. First, we use the underlying convex geometry of the Bayesian model to induce a Riemannian geometry on the parameter space. We then use differential geometry to identify the so-called natural or canonical loss function to perform Bayesian point estimation in that Riemannian manifold. For log-concave models, this canonical loss coincides with the Bregman divergence associated with the negative log posterior density. Following on from this, we show that the MAP estimator is the only Bayesian estimator that minimises the expected canonical loss, and that the posterior mean or MMSE estimator minimises the dual canonical loss. We then study the question of MAP and MSSE estimation performance in high dimensions. Precisely, we establish a universal bound on the expected canonical error as a function of image dimension, providing new insights the good empirical performance observed in convex problems. Together, these results provide a new understanding of MAP and MMSE estimation in log-concave settings, and of the multiple beneficial roles that convex geometry plays in imaging problems. Finally, we illustrate this new theory by analysing the regularisation-by-denoising Bayesian models, a class of state-of-the-art imaging models where priors are defined implicitly through image denoising algorithms, and an image denoising model with a wavelet shrinkage prior.

MAP Estimation With Bernoulli Randomness, and Its Application to Text Analysis and Recommender Systems

“…x t+1 := x t + at−xt t 7: end for estimation as an optimization problem. In some cases, the MAP problem (3) is a convex problem [4], then it can be solved well. In addition, in probabilistic models, we usually study the MAP problem in high dimensions.…”

“…It plays an essential role in various practical scenarios where there exist hidden variables or uncertainty. Some applications include image processing [3], [4], text analysis [5]- [7], recommender system [8], protein design and protein side-chain prediction problems [9], [10]. Adding the prior probability information reduces the overdependence on the observed data for parameter estimation, MAP estimation be seen as a regularization of Maximum Likelihood Estimation (MLE), MAP can deal well with low training data.…”

“…where P (D|x) denotes the likelihood of D, P (x) denotes the prior of x, and P (D) denotes the marginal probability of In general, the MAP problem is well-known to be NP-hard [11], [12], and is at least as computationally difficult as MLE [13]. Hence, there is a large number of studies about approximate methods [4], [9], [10], [14]- [20]. It is interesting that most existing studies focus on discrete MAP, i.e., when domain Ω is discrete.…”

“…In this work, we focus on the MAP problem which is continuous and non-convex, i.e., when the function −f (x) = − log P (D|x) − log P (x) is non-convex over the continuous domain Ω. In contrast with convex cases which are tractable [4], the non-convex MAP problem is NP-hard and hence intractable [11]. A popular choice is to employ some recent advances in non-convex optimization such as Concave-Convex procedure (CCCP) [21], Stochastic Majorization-Minimization (SMM) [22], Frank-Wolfe (FW) [23], Online Frank-Wolfe (OFW) [24], [25], Natasha2 [26], NEON2 [27] to solve the MAP estimation as a non-convex optimization problem.…”

Quantifying Uncertainty in High Dimensional Inverse Problems by Convex Optimisation

“…In the rest of this article we assume f (x) and g y (x) are closed convex functions which are not necessarily differentiable, and the objective functional (3) is computationally tractable with respect to x given the value of µ. For further details about MAP estimation see, e.g., [18]. A main computational advantage of the MAP estimator (3) is that it can be computed very efficiently, even in high dimensions, by using convex optimisation algorithms (e.g.…”