WPPNets and WPPFlows: The Power of Wasserstein Patch Priors for Superresolution

Abstract

We introduce WPPNets, which are CNNs trained by a new unsupervised loss function for image superresolution of materials microstructures. Instead of requiring access to a large database of registered high-and low-resolution images, we only assume to know a large database of low resolution images, the forward operator and one high-resolution reference image. Then, we propose a loss function based on the Wasserstein patch prior which measures the Wasserstein-2 distance between the patch distributions of the predictions and the reference image. We demonstrate by numerical examples that WPPNets outperform other methods with similar assumptions. In particular, we show that WPPNets are much more stable under inaccurate knowledge or perturbations of the forward operator. This enables us to use them in real-world applications, where neither a large database of registered data nor the exact forward operator are given.

PatchNR: Learning from Small Data by Patch Normalizing Flow Regularization

“…We compare our method with established methods from the literature. In particular, we compare with Wasserstein Patch Prior (WPP) [3,22], Expected Patch Log Likelihood (EPLL) [71] and Local Adversarial Regularizer (localAR) [50], as they work on patches and are unsupervised as well. Note that we optimize the EPLL GMM prior using a gradient descent optimizer, as half quadratic splitting proposed by the authors of [71] is much more expensive for the superresolution and CT forward operator.…”

“…For the experiments we consider material data which was also used in [3,22,24]. A series of multi-scale 3D images has been acquired by synchrotron micro-computed tomography at the SLS beamline TOMCAT.…”

“…Note that the KL divergence is not symmetric. Consequently, one could also derive a loss function for normalizing flows by interchanging the arguments of the KL divergence [3,20,37]. The loss function arising in this way is often called backward KL, while our approach is called forward KL.…”

“…• Wasserstein Patch Prior. The idea of the Wasserstein Patch Prior (WPP) [3,22] 10 is to use the Wasserstein-2 distance between the patch distribution of the reconstruction and the patch distribution of a given reference image. Here a high resolution reference image x (or a highresolution cutout) with a similar patch distribution as the unknown high resolution image x is needed.…”

“…Denoising methods based on the comparison of similar patches provided state-of-the-art methods [9,12,39,40] for a long time. Recently, the approximation of patch distributions of images was successfully exploited in certain papers [3,13,22,24,25,56]. In particular, the authors of [71] proposed the negative log likelihood of all patches of an image as a regularizer, where the underlying patch distribution was assumed to follow a Gaussian mixture model (GMM) which parameters were learned from few clean images.…”