Spectral Methods for Data Science: A Statistical Perspective

Abstract

Spectral methods have emerged as a simple yet surprisingly effective approach for extracting information from massive, noisy and incomplete data. In a nutshell, spectral methods refer to a collection of algorithms built upon the eigenvalues (resp. singular values) and eigenvectors (resp. singular vectors) of some properly designed matrices constructed from data. A diverse array of applications have been found in machine learning, data science, and signal processing, including recommendation systems, community detection, ranking, structured matrix recovery, tensor completion, joint shape matching, blind deconvolution, financial investments, risk managements, econometric modeling, treatment evaluations, causal inference, amongst others. Due to their simplicity and effectiveness, spectral methods are not only used as a stand-alone estimator, but also frequently employed to initialize other more sophisticated algorithms to improve performance.While the studies of spectral methods can be traced back to classical matrix perturbation theory and methods of moments, the past decade has witnessed tremendous theoretical advances in demystifying their efficacy through the lens of statistical modeling, with the aid of nonasymptotic random matrix theory. This monograph aims to present a systematic, comprehensive, yet accessible introduction to spectral methods from a modern statistical perspective, highlighting their algorithmic implications in diverse large-scale applications. In particular, our exposition gravitates around several central questions that span various applications: how to characterize the sample efficiency of spectral methods in reaching a target level of statistical accuracy, and how to assess their stability in the face of random noise, missing data, and adversarial corruptions? In addition to conventional 2 perturbation analysis, we present a systematic ∞ and 2,∞ perturbation theory for eigenspace and singular subspaces, which has only recently become available owing to a powerful "leave-one-out" analysis framework.

Inference for Heteroskedastic PCA with Missing Data

“…Low-rank matrix denoising serves as a common model to study the effectiveness of spectral methods (Chen et al, 2020c), and has been the main subject of many prior works including Abbe et al ( 2020 Lei (2019); ; Xia (2019b), among others. Several recent works began to pursue a distributional theory for the eigenvector or singular vectors of the observed data matrix (Bao et al, 2018;Cheng et al, 2020;Fan et al, 2020;Xia, 2019b).…”

“…This leaves us with two crucial terms to control, which forms the main content of this subsection. In particular, our proof relies heavily on the leave-one-out analysis framework that has proved effective in analyzing spectral methods (Abbe et al, 2020;Cai et al, 2021;Chen et al, 2020c).…”

“…In this section, we establish the auxiliary lemmas needed in the proof of Theorem 10. At the core of our analysis lies a leave-one-out (and leave-two-out) analysis framework that has been previously adopted to analyze spectral methods (Abbe et al, 2020;Cai et al, 2021;Chen et al, 2020cChen et al, , 2019b, which we shall introduce below.…”

The folded concave Laplacian spectral penalty learns block diagonal sparsity patterns with the strong oracle property

“…By another variant of the Davis-Kahan theorem, Lemma C.2, there exists an orthonormal matrix Θ ∈ R K×K such that ||V x − V y Θ|| op ≤ 2 3/2 P , so applying C.2 as above, Proof. of Lemma C.2 We will make frequent reference to standard matrix analysis results stated in Section 2 of (Chen et al, 2020). Let |||•||| be an arbitrary, unitarily invariant matrix norm; see Definition 2.6.1 of Chen et al (2020).…”

“…of Lemma C.2 We will make frequent reference to standard matrix analysis results stated in Section 2 of (Chen et al, 2020). Let |||•||| be an arbitrary, unitarily invariant matrix norm; see Definition 2.6.1 of Chen et al (2020).…”