Wasserstein Loss for Image Synthesis and Restoration

Abstract

This paper presents a novel variational approach to impose statistical constraints to the output of both image generation (to perform typically texture synthesis) and image restoration (for instance to achieve denoising and super-resolution) methods. The empirical distributions of linear or non-linear descriptors are imposed to be close to some input distributions by minimizing a Wasserstein loss, i.e. the optimal transport distance between the distributions. We advocate the use of a Wasserstein distance because it is robust when using discrete distributions without the need to resort to kernel estimators. We showcase different estimators to tackle various image processing applications. These estimators include linear wavelet-based filtering to account for simple textures, non-linear sparse coding coefficients for more complicated patterns, and the image gradient to restore sharper contents. For applications to texture synthesis, the input distributions are the empirical distributions computed from an exemplar image. For image denoising and super-resolution, the estimation process is more difficult; we propose to make use of parametric models and we show results using Generalized Gaussian Distributions.

Semi-discrete Optimal Transport in Patch Space for Enriching Gaussian Textures

“…Several tools from optimal transport (OT) have already been applied to texture synthesis: Rabin et al [20] and Xia et al [25] formulate texture mixing via Wasserstein barycenters, Tartavel et al [21] use Wasserstein distances in a variational formulation of texture synthesis, and Gutierrez et al [9] apply discrete OT in order to specify a global patch distribution. Here, we suggest to locally apply a semi-discrete transport plan which can be seen as a tweaked nearest-neighbor projection.…”

A Texture Synthesis Model Based on Semi-Discrete Optimal Transport in Patch Space

“…Wasserstein barycenters were used in [55,67] to address texture mixing. Tartavel et al [60] extended variational texture synthesis by combining discrete OT distances computed on several (non-linear) filter responses. Finally, Gutierrez et al [21] proposed a texture synthesis method that enforces the patch distribution at multiple scales by applying a discrete OT plan.…”

Texture synthesis guided by a low-resolution image

“…Recently, an approach has been proposed in [8,9], which introduces a preliminary step of dictionary learning for exploiting the given patch, and (not least) the Wasserstein distance for comparing the histograms of the entire texture with an extended version of the small patch. While the Wasserstein distance is well-known in image processing and computer vision under the name "earth mover distance" [10], it was only recently expanded to the context of texture synthesis [11,9,12]. Under the assumption that the main texture characteristics are contained in their Fourier magnitude [13], many works have shown that an efficient synthesis method is achieved when the texture phase is randomized [7,9].…”

“…Hence, we replicate the patch v so as to obtain a larger imageṽ ∈ R N such that the normalized histogram νṽ is equal to ν v . Although the Wasserstein distance is nonconvex (due to the histogram transformation), its gradient is Lipschitz-continuous and takes the following form [8,12]…”