Regularized Matrix Regression

Abstract

Summary Modern technologies are producing a wealth of data with complex structures. For instance, in two-dimensional digital imaging, flow cytometry and electroencephalography, matrix-type covariates frequently arise when measurements are obtained for each combination of two underlying variables. To address scientific questions arising from those data, new regression methods that take matrices as covariates are needed, and sparsity or other forms of regularization are crucial owing to the ultrahigh dimensionality and complex structure of the matrix data. The popular lasso and related regularization methods hinge on the sparsity of the true signal in terms of the number of its non-zero coefficients. However, for the matrix data, the true signal is often of, or can be well approximated by, a low rank structure. As such, the sparsity is frequently in the form of low rank of the matrix parameters, which may seriously violate the assumption of the classical lasso. We propose a class of regularized matrix regression methods based on spectral regularization. A highly efficient and scalable estimation algorithm is developed, and a degrees-of-freedom formula is derived to facilitate model selection along the regularization path. Superior performance of the method proposed is demonstrated on both synthetic and real examples.

svt: Singular Value Thresholding in MATLAB

“…The most direct approach for SVT is applying full SVD through svd and then soft-threshold the singular values. This approach is in practice used in many matrix learning problems according to the distributed code, e.g., Kalofolias, Bresson, Bronstein, and Vandergheynst (2014); Chi et al (2013); Parikh and Boyd (2013) ;Yang, Wang, Zhang, and Zhao (2013);Zhou, Liu, Wan, and Yu (2014); Zhou and Li (2014); Zhang et al (2017); Otazo, Candès, and Sodickson (2015); Goldstein, Studer, and Baraniuk (2015), to name a few. However, the built-in function svd is for full SVD of a dense matrix, and hence is very time-consuming and computationally expensive for large-scale problems.…”

“…The problem has sparked intensive research in recent years and is enjoying a broad range of applications such as personalized recommendation system (ACM SIGKDD and Netflix 2007) and imputation of massive genomics data (Chi, Zhou, Chen, Del Vecchyo, and Lange 2013). In matrix regression (Zhou and Li 2014), the predictors are two dimensional arrays such as images or measurements on a regular grid. Thus it requires a regression coefficient array of same size to completely capture the effects of matrix predictors.…”

“…Generic optimization methods such as accelerated proximal gradient algorithm, majorization-minorization (MM) algorithm, and alternating direction method of multipliers (ADMM) have been invoked to solve optimization problem (1). See, e.g., Mazumder et al (2010); Boyd, Parikh, Chu, Peleato, and Eckstein (2011); Parikh and Boyd (2013); Chi et al (2013); Lange, Chi, and Zhou (2014) for matrix completion algorithms and Zhou and Li (2014); Zhang et al (2017) for the accelerated proximal gradient method for solving nuclear norm penalized regression. All these algorithms involve repeated singular value thresholding, which is the proximal mapping associated with the nuclear norm regularization term…”

On the Degrees of Freedom of Mixed Matrix Regression

“…The true signal often has low rank, which can be well approximated by a low rank matrix. Recently, Zhou and Li [5] proposed the so-called regularized matrix regression model to deal with these matrix form data, which is based on spectral regularization. This model includes the well-known Lasso as a special case; see [8] for more details.…”

“…At the same time, the random noises are not always normal. These complex stochastic data are frequently collected in a large variety of research areas such as information technology, engineering, medical imaging and diagnosis, and finance [1][2][3][4][5][6][7]. For instance, a wellknown example is the study of an electroencephalography data set of alcoholism.…”