A Stochastic Multi-layer Algorithm for Semi-discrete Optimal Transport with Applications to Texture Synthesis and Style Transfer

Abstract

This paper investigates a new stochastic algorithm to approximate semi-discrete optimal transport for large-scale problem, i.e. in high dimension and for a large number of points. The proposed technique relies on a hierarchical decomposition of the target discrete distribution and the transport map itself. A stochastic optimization algorithm is derived to estimate the parameters of the corresponding multi-layer weighted nearest neighbor model. This model allows for fast evaluation during synthesis and training, for which it exhibits faster empirical convergence. Several applications to patch-based image processing are investigated: texture synthesis, texture inpainting, and style transfer. The proposed models compare favorably to the state of the art, either in terms of image quality, computation time, or regarding the number of parameters. Additionally, they do not require any pixel-based optimization or training on a large dataset of natural images.

A Wasserstein-Type Distance in the Space of Gaussian Mixture Models

“…More precisely, for instance for an image of size 256 × 256, the proposed approach takes about 35 seconds, whereas the semi-discrete approach of [16] takes about 400 seconds. We are currently exploring a multiscale version of this approach, inspired by the recent [22]. 8.…”

A Patch-Based Approach for Artistic Style Transfer Via Constrained Multi-Scale Image Matching

“…where Jk is the linear blending of J k (the output of the patch warping R k • W k+1 ↑ r ) and I k the content image at scale k. This linear combination is weighted (⊙ indicates pixel-wise multiplication) by a mask M k computed from the gradients of I k as done in [8]. This mask is clamped with the parameter ρ k ∈ [0, 1] which controls the amount of injected geometric details from I k .…”

“…This one can build new patterns from the patches of the style image even in plain regions. [8] introduces a multi-layer algorithm for semi-discrete optimal transport, and applies it in the patch space in the context of both texture synthesis and style transfer. This method then matches the patch distribution of the style image, while the content features are blended based on a edge detection.…”

Nearly Tight Convergence Bounds for Semi-discrete Entropic Optimal Transport

“…In numerical analysis, the semi-discrete setting gives a natural framework to approximate the solution of the optimal transport problem between a probability density ρ and a probability measure µ that consists in approximating µ by a sequence of measures (µ N ) N ≥1 with finite support such that lim N →+∞ W 2 (µ, µ N ) = 0 (Oliker & Prussner, 1989;Cullen et al, 1991;Gangbo & McCann, 1996;Caffarelli et al, 1999;Mirebeau, 2015). Finally in image processing, semi-discrete transport has proved useful for texture synthesis and style transfer (Galerne, Leclaire, & Rabin, 2017, 2018Leclaire & Rabin, 2020). We thus focus in this work on the semi-discrete setting, and show that we can improve the recent asymptotic bounds given in (Altschuler et al, 2021) under slightly stronger regularity assumptions on the source measure.…”