Riemannian Langevin Algorithm for Solving Semidefinite Programs

Abstract

We propose a Langevin diffusion-based algorithm for non-convex optimization and sampling on a product manifold of spheres. Under a logarithmic Sobolev inequality, we establish a guarantee for finite iteration convergence to the Gibbs distribution in terms of Kullback-Leibler divergence. We show that with an appropriate temperature choice, the suboptimality gap to the global minimum is guaranteed to be arbitrarily small with high probability.As an application, we analyze the proposed Langevin algorithm for solving the Burer-Monteiro relaxation of a semidefinite program (SDP). In particular, we establish a logarithmic Sobolev inequality for the Burer-Monteiro problem when there are no spurious local minima; hence implying a fast escape from saddle points. Combining the results, we then provide a global optimality guarantee for the SDP and the Max-Cut problem. More precisely, we show the Langevin algorithm achieves -multiplicative accuracy with high probability in Ω(n 2 −3 ) iterations, where n is the size of the cost matrix. Contents

Analysis of Langevin Monte Carlo from Poincaré to Log-Sobolev

“…Our work is inspired by [VW19], which advocated the use of a functional inequality paired with a smoothness condition as a minimal set of assumptions for obtaining sampling guarantees; in their work, Vempala and Wibisono prove convergence of LMC under a log-Sobolev inequality. This result was then improved using the proximal Langevin algorithm under higher-order smoothness in [Wib19] and extended to Riemannian manifolds in [LE20].…”

Improved Convergence Rate of Stochastic Gradient Langevin Dynamics with Variance Reduction and its Application to Optimization

“…In this appendix, we prove Property C.4 which is only a slight adaptation of Theorem 3.4 from Li and Erdogdu (2020). We show that with additional weak Morse, and smoothness assumptions to dissipativity, we can obtain a Log-Sobolev constant of dν ∝ e −γF dx whose inverse only depends linearly on the inverse temperature parameter.…”

“…Q.E.D Lemma D.2 (Li and Erdogdu (2020), Proposition E.5). Let W t and Wt be weak solutions on some filtered probability space of the following one dimensional SDE's:…”

“…On the other hand, under the additional condition of weak Morse, Lipschitzness of ∇ 2 F and other minor assumptions, we can obtain a far better Log-Sobolev constant whose inverse depends only linearly on γ as follows. This is a straightforward adaptation of Li and Erdogdu's result (Li and Erdogdu, 2020). We provide a proof in Appendix D. Property C.4.…”

“…We also prove the convergence of SVRG-LD and SARAH-LD to the global minimum under an additional assumption of dissipativity with a gradient complexity of Õ((n + n 1/2 ǫ −1 dLα −1 )γ 2 L 2 α −2 ) which is better than previous work since it has almost all the time a dependence on n of O( √ n) and does not require the batch size and the inner loop length to depend on the accuracy ǫ. On the other hand, we import the idea of Li and Erdogdu (2020) from product manifolds of spheres to the Euclidean space in order to show that under the additional assumption of weak Morse, the convergence in the Euclidean space can be accelerated by eliminating the exponential dependence on 1/ǫ.…”