An extension of fast iterative shrinkage‐thresholding algorithm to Riemannian optimization for sparse principal component analysis

Abstract

Sparse principal component analysis (PCA), an important variant of PCA, attempts to find sparse loading vectors when conducting dimension reduction. This paper considers the nonsmooth Riemannian optimization problem associated with the ScoTLASS model1 for sparse PCA which can impose orthogonality and sparsity simultaneously. A Riemannian proximal method is proposed in the work of Chen et al.9 for the efficient solution of this optimization problem. In this paper, two acceleration schemes are introduced. First and foremost, we extend the FISTA method from the Euclidean space to the Riemannian manifold to solve sparse PCA, leading to the accelerated Riemannian proximal gradient method. Since the Riemannian optimization problem for sparse PCA is essentially nonconvex, a restarting technique is adopted to stabilize the accelerated method without sacrificing the fast convergence. Second, a diagonal preconditioner is proposed for the Riemannian proximal subproblem which can further accelerate the convergence of the Riemannian proximal methods. Numerical evaluations establish the computational advantages of the proposed methods over the existing proximal gradient methods on a manifold. Additionally, a short result concerning the convergence of the Riemannian subgradients of a sequence is established, which, together with the result in the work of Chen et al.,9 can show the stationary point convergence of the Riemannian proximal methods.

An image inpainting algorithm using exemplar matching and low-rank sparse prior

“…• An inexact accelerated Riemannian proximal gradient method (IARPG) is used to solve the optimization problem from the proposed model. IARPG merges the Nesterov momentum acceleration technique for Riemannian optimization [HW21b] and the inexact Riemannian proximal gradient method (RPG) in [HW21c]. Note that though the existing RPGs are designed for general manifold, it has not been used for the manifold of fixed-rank matrices before.…”

Riemannian proximal gradient methods

“…where A ∈ R m×n is the data matrix. This model is a penalized version of the ScoTLASS model introduced in [JTU03] and it has been used in [CMSZ20,HW21b].…”

“…where M is a finite dimensional Riemannian manifold. Such optimization problem is of interest due to many important applications including but not limit to compressed models [OLCO13], sparse principal component analysis [ZHT06,HW21b], sparse variable principal component analysis [US08, CMW13, XLY20], discriminative k-means [YZW08], texture and imaging inpainting [LRZM12], cosparse factor regression [MDC17], and low-rank sparse coding [ZGL + 13, SQ16].…”

“…It is shown that such proximal mapping can be solved efficiently by a semi-smooth Newton method when the manifold M is the Stiefel manifold. In [HW21b], a diagonal weighted proximal mapping is defined by replacing η 2 F in (1.4) with η, W η F , where the diagonal weighted linear operator W is carefully selected. Moreover, the Nesterov momentum acceleration technique is further introduced to accelerate the algorithm, yielding an algorithm called AManPG.…”

“…The convergence analyses of Riemannian proximal gradient methods in [CMSZ20, HW21b,HW21a] all rely on solving proximal mappings (1.4) and (1.5) exactly. On the one hand, solving these proximal mappings exactly is not practicable due to rounding errors from the finite precision arithmetic.…”