Majorization-Minimization on the Stiefel Manifold With Application to Robust Sparse PCA

A Semidefinite Relaxation for Sums of Heterogeneous Quadratics on the Stiefel Manifold

“…In practical settings for high-dimensional data, a variety of iterative local methods are often applied to solve nonconvex problems over the Stiefel manifold, from gradient ascent by geodesics [2; 18; 1] and more recently majorization-minimization (MM) algorithms, where Breloy et al [14] applied MM methods to solve (1) with guarantees of convergence to a stationary point. While the computational complexity and memory requirements of these solvers scale well, their obtained solutions lack any global optimality guarantees.…”

“…Using [4,Section 6.6.3] it can be shown that computing the certificate only, with a given Ū , results in a substantial reduction in flops by a factor of O(d 3 /k) over solving (SDP-D). Subsequently, a first-order MM solver [14], whose cost is O(dk 2 + k 3 ) per iteration, combined with our global optimality certificate, is an obvious preference to solving the full SDP in (SDP-P) for large problems. See Appendix D.1 for more details.…”

“…In this section, we use the Stiefel majorization-minimization (StMM) solver with a linear majorizer from Breloy et al [14] to obtain a local solution ŪMM to (1) for various inputs M i and use Theorem 4.1 to certify the local solution either as globally optimal or as a non-optimal stationary point. For comparison, we obtain candidate solutions Xi from the SDP and perform a rank-one SVD of each to form ŪSDP , i.e.…”

“…where St(k, d) = {U ∈ R d×k : U U = I k } denotes the Stiefel manifold. This problem arises in modern signal processing and machine learning applications like heteroscedastic probabilistic principal component analysis (HPPCA) [24], heterogeneous clutter in radar sensing [38], and robust sparse PCA [14]. Each of these applications involves learning a signal subspace for data possessing heterogeneous statistics.…”

An iterative coordinate descent algorithm to compute sparse low-rank approximations

“…Following this work, several LASSO and convex optimization (particularly semidefinite programming) approaches were developed to deal with the sparsity of the singular vectors [11], [12], [13], [14]. There are numerous ways to deal with the sPCA and these include approaches such as: greedy methods [15], geodesic steepest descent [16], Givens rotations [17], low rank approximations via a regularized (sparsity promoting) singular value decomposition [18], truncated power iterations [19], [20], [21], steepest descent on the Stiefel manifold using rotations matrices [22] or on the Grassmannian manifold [23], quasi-Newton optimization for the sparse generalized eigenvalue problem [24], iterative deflation techniques [25], the minorization-maximization framework [26] with additional orthogonality constraints [27] or on the Stiefel manifold [28],…”