Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates

Abstract

Stochastic majorization-minimization (SMM) is an online extension of the classical principle of majorization-minimization, which consists of sampling i.i.d. data points from a fixed data distribution and minimizing a recursively defined majorizing surrogate of an objective function. In this paper, we introduce stochastic block majorization-minimization, where the surrogates can now be only block multi-convex and a single block is optimized at a time within a diminishing radius. Relaxing the standard strong convexity requirements for surrogates in SMM, our framework gives wider applicability including online CANDECOMP/PARAFAC (CP) dictionary learning and yields greater computational efficiency especially when the problem dimension is large. We provide an extensive convergence analysis on the proposed algorithm, which we derive under possibly dependent data streams, relaxing the standard i.i.d. assumption on data samples. We show that the proposed algorithm converges almost surely to the set of stationary points of a nonconvex objective under constraints at a rate O((log n) 1+ε /n 1/2 ) for the empirical loss function and O((log n) 1+ε /n 1/4 ) for the expected loss function, where n denotes the number of data samples processed. Under some additional assumption, the latter convergence rate can be improved to O((log n) 1+ε /n 1/2 ). Our results provide first convergence rate bounds for various online matrix and tensor decomposition algorithms under a general Markovian data setting.

Convergence and Complexity of Stochastic Subgradient Methods with Dependent Data for Nonconvex Optimization

“…8.1. This improves the rate of convergence of stochastic algorithms for constrained nonconvex expected loss minimization with dependent data [Lyu22], see Thm. 8.1 for the details.…”

“…Stochastic (sub)gradient descent (SGD) is also recently considered in [WPT + 21] for convex problems. In the constrained nonconvex case, the work [LNB20] showed asymptotic guarantees of stochastic majorization-minimization (SMM)-type algorithms to stationary points of the expected loss function and the recent work [Lyu22] showed nonasymptotic guarantees. More recently, [Lyu22] studied a generalized SMM-type algorithms and showed the complexity Õ(ε −8 ) in the general case and Õ(ε −4 ) when all the iterates of the algorithm lie in the interior of the constraint set, for making the stationarity gap (see LHS of (3)) for the expected loss function less than ε.…”

“…In the constrained nonconvex case, the work [LNB20] showed asymptotic guarantees of stochastic majorization-minimization (SMM)-type algorithms to stationary points of the expected loss function and the recent work [Lyu22] showed nonasymptotic guarantees. More recently, [Lyu22] studied a generalized SMM-type algorithms and showed the complexity Õ(ε −8 ) in the general case and Õ(ε −4 ) when all the iterates of the algorithm lie in the interior of the constraint set, for making the stationarity gap (see LHS of (3)) for the expected loss function less than ε. We also remark that [Lyu22] showed that for the 'empirical loss functions' (recursive average of sample losses), SMM-type algorithms need only Õ(ε −4 ) iterations for making the stationarty gap under ε.…”

“…More recently, [Lyu22] studied a generalized SMM-type algorithms and showed the complexity Õ(ε −8 ) in the general case and Õ(ε −4 ) when all the iterates of the algorithm lie in the interior of the constraint set, for making the stationarity gap (see LHS of (3)) for the expected loss function less than ε. We also remark that [Lyu22] showed that for the 'empirical loss functions' (recursive average of sample losses), SMM-type algorithms need only Õ(ε −4 ) iterations for making the stationarty gap under ε. Our present work does not consider complexity with respect to the empirical loss functions.…”

“…Our present work does not consider complexity with respect to the empirical loss functions. See [Lyu22] for more details.…”