Spectral Universality of Regularized Linear Regression with Nearly Deterministic Sensing Matrices

Abstract

It has been observed that the performances of many high-dimensional estimation problems are universal with respect to underlying sensing (or design) matrices. Specifically, matrices with markedly different constructions seem to achieve identical performance if they share the same spectral distribution and have "generic" singular vectors. We prove this universality phenomenon for the case of convex regularized least squares (RLS) estimators under a linear regression model with additive Gaussian noise. Our main contributions are two-fold: (1) We introduce a notion of universality classes for sensing matrices, defined through a set of deterministic conditions that fix the spectrum of the sensing matrix and precisely capture the previously heuristic notion of generic singular vectors; (2) We show that for all sensing matrices that lie in the same universality class, the dynamics of the proximal gradient descent algorithm for solving the regression problem, as well as the performance of RLS estimators themselves (under additional strong convexity conditions) are asymptotically identical. In addition to including i.i.d. Gaussian and rotational invariant matrices as special cases, our universality class also contains highly structured, strongly correlated, or even (nearly) deterministic matrices. Examples of the latter include randomly signed versions of incoherent tight frames and randomly subsampled Hadamard transforms. As a consequence of this universality principle, the asymptotic performance of regularized linear regression on many structured matrices constructed with limited randomness can be characterized by using the rotationally invariant ensemble as an equivalent yet mathematically more tractable surrogate.

Bayes-optimal limits in structured PCA, and how to reach them

“…This intuition comes from a very recent line of work concerning linear regression and phase retrieval with structured matrices of covariates. Indeed, the authors of [37,38,39] show that in this context, the class of rotationally invariant matrices leads to the same performance as a much broader class of almost deterministic matrices (with the same spectral density), also when AMP or its linearized version are used as inference algorithm. This is a different setting from the one we consider, since in our setup the structured matrix is the noise, but it nevertheless suggests that our predictions should remain true more generically.…”

“…The most tricky parameter is M , that we introduced to decouple the four body interactions in the Hamiltonian. Notice first that (recall definitions (39) and ( 41))…”