Optimal Combination of Linear and Spectral Estimators for Generalized Linear Models

Abstract

We study the problem of recovering an unknown signal $${\varvec{x}}$$ x given measurements obtained from a generalized linear model with a Gaussian sensing matrix. Two popular solutions are based on a linear estimator $$\hat{\varvec{x}}^\mathrm{L}$$ x ^ L and a spectral estimator $$\hat{\varvec{x}}^\mathrm{s}$$ x ^ s . The former is a data-dependent linear combination of the columns of the measurement matrix, and its analysis is quite simple. The latter is the principal eigenvector of a data-dependent matrix, and a recent line of work has studied its performance. In this paper, we show how to optimally combine $$\hat{\varvec{x}}^\mathrm{L}$$ x ^ L and $$\hat{\varvec{x}}^\mathrm{s}$$ x ^ s . At the heart of our analysis is the exact characterization of the empirical joint distribution of $$({\varvec{x}}, \hat{\varvec{x}}^\mathrm{L}, \hat{\varvec{x}}^\mathrm{s})$$ ( x , x ^ L , x ^ s ) in the high-dimensional limit. This allows us to compute the Bayes-optimal combination of $$\hat{\varvec{x}}^\mathrm{L}$$ x ^ L and $$\hat{\varvec{x}}^\mathrm{s}$$ x ^ s , given the limiting distribution of the signal $${\varvec{x}}$$ x . When the distribution of the signal is Gaussian, then the Bayes-optimal combination has the form $$\theta \hat{\varvec{x}}^\mathrm{L}+\hat{\varvec{x}}^\mathrm{s}$$ θ x ^ L + x ^ s and we derive the optimal combination coefficient. In order to establish the limiting distribution of $$({\varvec{x}}, \hat{\varvec{x}}^\mathrm{L}, \hat{\varvec{x}}^\mathrm{s})$$ ( x , x ^ L , x ^ s ) , we design and analyze an approximate message passing algorithm whose iterates give $$\hat{\varvec{x}}^\mathrm{L}$$ x ^ L and approach $$\hat{\varvec{x}}^\mathrm{s}$$ x ^ s . Numerical simulations demonstrate the improvement of the proposed combination with respect to the two methods considered separately.

Subspace clustering in high-dimensions: Phase transitions \& Statistical-to-Computational gap

“…The initialization strategy we considered in the uninformed case for Algorithm 1 is not the only possible choice, there are smarter ways of initialising which can lead to a considerable improvement without explicitly assuming any information about the signal, e.g. spectral initialization [35]. Looking at Fig.…”

Exponentially Fast Concentration of Vector Approximate Message Passing to its State Evolution

“…Gaussian measurement matrices, have found many applications in the literature to date. These algorithms play an important role in high-dimensional statistics [10,12,41], wireless communications [7,22], and many other applications [1,30,31,32,42,44]. In particular, the finite sample analysis for VAMP and GAMP presented here can be used to find the error rates for the capacityachieving sparse regression coding schemes introduced in [7,22], which use VAMP and GAMP as decoders.…”

“…In particular, there has been no concentration results developed for generalized versions of AMP, which can handle models such as (1.1), or for AMP algorithms with measurement matrices that are not i.i.d. Gaussian, despite such algorithms playing an important role in high-dimensional statistics [10,12,41], wireless communications [7,22], and many other applications [1,30,31,32,42,44].…”

Spectral Universality of Regularized Linear Regression with Nearly Deterministic Sensing Matrices

“…Such asymptotic results are highly valuable, as they provide fundamental limits on the degree to which different inference methodology can be successful. Moreover, the precise asymptotic characterizations can also lead to optimal algorithm designs, as demonstrated in recent work [13,24,53,62,64,70,89].…”