Bridging Bayesian and Minimax Mean Square Error Estimation via Wasserstein Distributionally Robust Optimization

Abstract

We introduce a distributionally robust minimium mean square error estimation model with a Wasserstein ambiguity set to recover an unknown signal from a noisy observation. The proposed model can be viewed as a zero-sum game between a statistician choosing an estimator—that is, a measurable function of the observation—and a fictitious adversary choosing a prior—that is, a pair of signal and noise distributions ranging over independent Wasserstein balls—with the goal to minimize and maximize the expected squared estimation error, respectively. We show that, if the Wasserstein balls are centered at normal distributions, then the zero-sum game admits a Nash equilibrium, by which the players’ optimal strategies are given by an affine estimator and a normal prior, respectively. We further prove that this Nash equilibrium can be computed by solving a tractable convex program. Finally, we develop a Frank–Wolfe algorithm that can solve this convex program orders of magnitude faster than state-of-the-art general-purpose solvers. We show that this algorithm enjoys a linear convergence rate and that its direction-finding subproblems can be solved in quasi-closed form.

Wasserstein Distributionally Robust Gaussian Process Regression and Linear Inverse Problems

“…++ is invertible. Similar finite-dimensional duality results have been established in [22,24], but with a slight difference in the definition of the Wasserstein distance and the ambiguity set. The rest of the proof then argues that the error of approximations are negligible as the dimension grows to infinity.…”

Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning

“…In some applications one has additional structural information about the relation between the signal x and the observation y (e.g., the measurement noise may be known to be independent of the signal, or the observation may be governed by a linear measurement model, etc.). Such structural information can be used to restrict the Wasserstein ambiguity set in (35), thereby reducing the conservativeness of the distributionally robust MMSE estimator [69].…”

On a Stochastic Fundamental Lemma and Its Use for Data-Driven Optimal Control

“…The most commonly used ambiguity sets employ the Wasserstein metric. However, tractable reformulations of Wasserstein ambiguity sets are limited to certain empirical distributions [1] or to ambiguity sets comprising Gaussians [12]. As an alternative, Gelbrich ambiguity sets include all distributions with moments that closely match a given empirical pair ( m, Γ) based on the Gelbrich distance in Definition 1.…”