Gradient descent with non-convex constraints: local concavity determines convergence

Abstract

Many problems in high-dimensional statistics and optimization involve minimization over non-convex constraints—for instance, a rank constraint for a matrix estimation problem—but little is known about the theoretical properties of such optimization problems for a general non-convex constraint set. In this paper we study the interplay between the geometric properties of the constraint set and the convergence behavior of gradient descent for minimization over this set. We develop the notion of local concavity coefficients of the constraint set, measuring the extent to which convexity is violated, which governs the behavior of projected gradient descent over this set. We demonstrate the versatility of these concavity coefficients by computing them for a range of problems in low-rank estimation, sparse estimation and other examples. Through our understanding of the role of these geometric properties in optimization, we then provide a convergence analysis when projections are calculated only approximately, leading to a more efficient method for projected gradient descent in low-rank estimation problems.

Recovery of simultaneous low rank and two-way sparse coefficient matrices, a nonconvex approach

“…Compared to the classical nonconvex optimization theory, which only shows a sublinear convergence to a local optima, the focus of the recent literature is on establishing linear rates of convergence or characterizing that the objective does not have spurious local minima. In addition to the methods that work on the factorized form, [63,76,64,13] consider projected gradient-type methods which optimize over the matrix variable Θ ∈ R m 1 ×m 2 . These methods involve calculating the top r singular vectors of an m 1 × m 2 matrix at each iteration.…”

“…Taking average, and plugging in the main result (13) and the statistical error (19) we obtain our desired result.…”

“…From a practical point of view, it has been observed that directly working with a nonconvex optimization problem can lead to both faster and more accurate algorithms [104,133,127,117]. As a result, a body of literature has recently emerged that tries to characterize good performance of these algorithms [13,132,52].…”

Hypercluster: a flexible tool for parallelized unsupervised clustering optimization

“…Researchers therefore often compare results from multiple algorithms and hyperparameters [ 7 , 24 – 28 ]. Typically, the effect of hyperparameter choice on the quality of clustering results cannot be described with a convex function, meaning that hyperparameters should be chosen through exhaustive grid search [ 29 ], a slow and cumbersome process. Software packages for automatic hyperparameter tuning and model selection for regression and classification exist, notably auto-sklearn from AutoML [ 30 ], and some groups have made excellent tools for distributing a single clustering calculation for huge datasets [ 31 , 32 ], but to the best of our knowledge, there is no package for comparing several clustering algorithms and hyperparameters.…”

“…hyperparameters should be chosen through exhaustive grid search [29], a slow and cumbersome process. Software packages for automatic hyperparameter tuning and model selection for regression and classification exist, notably auto-sklearn from AutoML [30], and some groups have made excellent tools for distributing a single clustering calculation for huge datasets [31,32], but to the best of our knowledge, there is no package for comparing several clustering algorithms and hyperparameters.…”