Optimizing over the Growing Spectrahedron

Abstract

We devise a framework for computing an approximate solution path for an important class of parameterized semidefinite problems that is guaranteed to be ε-close to the exact solution path. The problem of computing the entire regularization path for matrix factorization problems such as maximum-margin matrix factorization fits into this framework, as well as many other nuclear norm regularized convex optimization problems from machine learning. We show that the combinatorial complexity of the approximate path is independent of the size of the matrix. Furthermore, the whole solution path can be computed in near linear time in the size of the input matrix. The framework employs an approximative semidefinite program solver for a fixed parameter value. Here we use an algorithm that has recently been introduced by Hazan. We present a refined analysis of Hazan's algorithm that results in improved running time bounds for a single solution as well as for the whole solution path as a function of the approximation guarantee.

New analysis and results for the Frank–Wolfe method

“…As early as 1970, Demyanov and Rubinov [3] used the FW gap quantities extensively in their convergence proofs of the Frank-Wolfe method, and perhaps this quantity was used even earlier. In certain contexts, G k is an important quantity by itself, see for example Hearn [14], Khachiyan [16] and Giesen et al [11]. Indeed, Hearn [14] studies basic properties of the FW gaps independent of their use in any algorithmic schemes.…”

“…Indeed, Hearn [14] studies basic properties of the FW gaps independent of their use in any algorithmic schemes. For results concerning upper bound guarantees on G k for specific and general problems see Khachiyan [16], Clarkson [1], Hazan [13], Jaggi [15], Giesen et al [11], and Harchaoui et al [12]. Both B w k and G k are computed directly from the solution of the linear optimization problem in step (2.)…”

“…(Indeed, in the minmax setting notice that the optimal response x k in (6) is a function of the current iterate λ k and hence f (x k ) − h(λ k ) = B m k − h(λ k ) is not just any duality gap but rather is determined completely by the current iterate λ k . This special feature of the duality gap B m k − h(λ k ) is exploited in the application of the Frank-Wolfe method to rounding of polytopes [16], parametric optimization on the spectrahedron [11], and to regularized regression [10] (and perhaps elsewhere as well), where bounds on the FW gap G k are used to bound B m k − h(λ k ) directly.) We also mention that in some applications there might be exact knowledge of the optimal value h * , such as in certain linear regression and/or machine learning applications where one knows a priori that the optimal value of the loss function is zero.…”

“…The original Frank-Wolfe method, developed for smooth convex optimization on a polytope, dates back to Frank and Wolfe [9], and was generalized to the more general smooth convex objective function over a bounded convex feasible region thereafter, see for example Demyanov and Rubinov [3], Dunn and Harshbarger [8], Dunn [6], [7], also Levitin and Polyak [19] and Polyak [24]. More recently there has been renewed interest in the Frank-Wolfe method due to some of its properties that we will shortly discuss, see for example Clarkson [1], Hazan [13], Jaggi [15], Giesen et al [11], and most recently Harchaoui et al [12], Lan [18] and Temlyakov [25]. The Frank-Wolfe method is premised on being able to easily solve (at each iteration) linear optimization problems over the feasible region of interest.…”

Fast and scalable Lasso via stochastic Frank–Wolfe methods with a convergence guarantee

“…The key observation is that, when Σ is a polytope (e.g. the unit simplex for L 2 -SVMs [35], the ℓ 1 -ball of radius δ for the Lasso problem (1), a spectrahedron in nuclear norm for matrix recovery [14]), the search in step 3 can be reduced to a search among the vertices of Σ. This allows to devise cheap analytical formulas to find u (k) , ensuring that each iteration has an overall cost of O(p).…”