Block-Randomized Stochastic Proximal Gradient for Low-Rank Tensor Factorization

Abstract

This work considers the problem of computing the canonical polyadic decomposition (CPD) of large tensors. Prior works mostly leverage data sparsity to handle this problem, which is not suitable for handling dense tensors that often arise in applications such as medical imaging, computer vision, and remote sensing. Stochastic optimization is known for its low memory cost and per-iteration complexity when handling dense data. However, exisiting stochastic CPD algorithms are not flexible enough to incorporate a variety of constraints/regularizations that are of interest in signal and data analytics. Convergence properties of many such algorithms are also unclear. In this work, we propose a stochastic optimization framework for large-scale CPD with constraints/regularizations. The framework works under a doubly randomized fashion, and can be regarded as a judicious combination of randomized block coordinate descent (BCD) and stochastic proximal gradient (SPG). The algorithm enjoys lightweight updates and small memory footprint. In addition, this framework entails considerable flexibility-many frequently used regularizers and constraints can be readily handled under the proposed scheme. The approach is also supported by convergence analysis. Numerical results on large-scale dense tensors are employed to showcase the effectiveness of the proposed approach.

Multi-Way Analysis Coupled with Near-Infrared Spectroscopy in Food Industry: Models and Applications

“…In order to improve the convergence speed of the conventional PARAFAC-ALS algorithm, a wide range of remedies have been developed, such as line search [ 52 ], enhanced line search [ 53 ], extrapolation with optimized step size and search direction [ 54 ] and compression [ 46 ]. Besides the ALS algorithm, some alternatives for calculating PARAFAC models have also been proposed, including the all-at-once algorithm [ 55 ], the hierarchical conjugate gradient algorithm [ 56 ], a random gradient algorithm [ 57 , 58 ], the fast damped Gauss–Newton algorithm [ 52 ], etc. These new alternatives were reported to have better convergence speed for many real cases.…”

Fiber-Sampled Stochastic Mirror Descent for Tensor Decomposition with β-Divergence

“…In recent year, a fiber sampling [14,19,21] strategy was used in Euclidean loss based tensor decomposition and completion to reduce the complexity and memory burdens. In [14,19], fiber samplingbased stochastic CPD algorithms select samples i that are related to a single latent factor An and updates An based on the selected fiber samples per iteration. In the context of β-divergence-based CPD, the sampling strategy admits a couple of notable advantages:…”

“…• Incorporating Prior on An: Randomly sampling some indexes i [22] or selecting a subtensor [13] faces an issue that samples may relate to only parts of An. Useful prior information about the entire latent factor (e.g., column norm constraints) cannot be imposed; see [14,15] for more discussion. Fiber sampling does not have this challenge.…”

“…Designing CPD algorithms under the least squares loss has been a central task in the tensor community, for which effective first-order, (quasi-)second-order, and stochastic optimization algorithms were all developed; see, e.g., [11][12][13][14]. More discussions on algorithms can be found in recent review paper [15].…”

“…In this paper, by exploiting the multilinear algebraic structure of low-rank tensors, a unified stochastic mirror descent (MD) algorithmic framework is developed for large-scale β-divergence CPD. Our idea is to integrate a recently proposed tensor fiber sampling strategy [14,19] with the MD algorithm. The fiber sampling strategy gives rise to nicely structured (non-)convex subproblems with respect to the latent factors under the β-divergence, which admits simple and efficient mirror descent-based updates.…”