The Deep Minimizing Movement Scheme

Hwang, Hyung Ju; Kim, Cheolhyeong; Park, Min Sue; Son, Hwijae

doi:10.48550/arxiv.2109.14851

Cited by 3 publications

(6 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pushing forward a probability distribution by the proximal operator corresponds to one step of the JKO scheme for Wasserstein gradient flow of a linear functional in the space of distributions [Jordan et al, 1998, Benamou et al, 2016. Compared to recent works on neural Wasserstein gradient flow [Mokrov et al, 2021, Hwang et al, 2021, Bunne et al, 2021, where a separate network is needed to parameterize the pushforward map for every JKO step, our linear functional yields a pushforward map that is identical for each step; this property allows us to use a single neural network as a parameterization.…”

Section: Related Workmentioning

confidence: 99%

Learning Proximal Operators to Discover Multiple Optima

Li¹,

Aigerman²,

Kim³

et al. 2022

Preprint

View full text Add to dashboard Cite

Finding multiple solutions of non-convex optimization problems is a ubiquitous yet challenging task. Typical existing solutions either apply single-solution optimization methods from multiple random initial guesses or search in the vicinity of found solutions using ad hoc heuristics. We present an end-to-end method to learn the proximal operator across a family of non-convex problems, which can then be used to recover multiple solutions for unseen problems at test time. Our method only requires access to the objectives without needing the supervision of ground truth solutions. Notably, the added proximal regularization term elevates the convexity of our formulation: by applying recent theoretical results, we show that for weakly-convex objectives and under mild regularity conditions, training of the proximal operator converges globally in the over-parameterized setting. We further present a benchmark for multi-solution optimization including a wide range of applications and evaluate our method to demonstrate its effectiveness.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning Proximal Operators to Discover Multiple Optima

Li¹,

Aigerman²,

Kim³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The time and spatial discretizations of Wasserstein gradient flows are extensively studied in literature (Jordan et al, 1998;Junge et al, 2017;Carrillo et al, 2021a,b;Bonet et al, 2021;Liutkus et al, 2019;Frogner & Poggio, 2020). Recently, neural networks have been applied in solving or approximating Wasserstein gradient flows (Mokrov et al, 2021;Lin et al, 2021b,a;Alvarez-Melis et al, 2021;Bunne et al, 2021;Hwang et al, 2021;Fan et al, 2021). For sampling algorithms, di Langosco et al (2021) learns the transportation function by solving an unregularized variational problem in the family of vector-output deep neural networks.…”

Section: Introductionmentioning

confidence: 99%

Optimal Neural Network Approximation of Wasserstein Gradient Direction via Convex Optimization

Wang¹,

Chen²,

Pilancı³

et al. 2022

Preprint

View full text Add to dashboard Cite

The computation of Wasserstein gradient direction is essential for posterior sampling problems and scientific computing. The approximation of the Wasserstein gradient with finite samples requires solving a variational problem. We study the variational problem in the family of two-layer networks with squared-ReLU activations, towards which we derive a semi-definite programming (SDP) relaxation. This SDP can be viewed as an approximation of the Wasserstein gradient in a broader function family including two-layer networks. By solving the convex SDP, we obtain the optimal approximation of the Wasserstein gradient direction in this class of functions. Numerical experiments including PDE-constrained Bayesian inference and parameter estimation in COVID-19 modeling demonstrate the effectiveness of the proposed method.

show abstract

“…Very recently, efforts have been made in using NNs to solve evolution PDEs [7,14,30]. In these approaches, instead of approximating the solution of the evolution PDE over the the whole time-space domain, the NNs are used to represent the solution with time-dependent parameters.…”

Section: Introductionmentioning

confidence: 99%

“…We note that the numerical algorithm to solve the generalized diffusion (5) can also be formulated in the Eulerian frame of reference based the notion of Wasserstein gradient flow [32]. We refer the interested readers to a recent work [30] for a neural network implementation of the minimizing movement scheme with the Wasserstein distance. Since gradient flows also provide a continuous variational formulation of many machine learning algorithms [18,62], the numerical approaches developed here will have wide applications in many machine learning tasks, such as supervised learning [18], variational inference [62], density estimation [60] and generative models [29].…”

Section: Introductionmentioning

confidence: 99%