The Noise-Sensitivity Phase Transition in Compressed Sensing

Abstract

Consider the noisy underdetermined system of linear equations: y = Ax 0 + z 0 , with n × N measurement matrix A, n < N , and Gaussian white noise z 0 ∼ N(0, σ 2 I). Both y and A are known, both x 0 and z 0 are unknown, and we seek an approximation to x 0 . When x 0 has few nonzeros, useful approximations are often obtained by 1 -penalized 2 minimization, in which the reconstructionx 1,λ solves min y − Ax 2 2 /2 + λ x 1 . Evaluate performance by mean-squared error (MSE = E||xConsider matrices A with iid Gaussian entries and a large-system limit in which n, N → ∞ with n/N → δ and k/n → ρ. Call the ratio MSE/σ 2 the noise sensitivity. We develop formal expressions for the MSE ofx 1,λ , and evaluate its worst-case formal noise sensitivity over all types of k-sparse signals. The phase space 0 ≤ δ, ρ ≤ 1 is partitioned by curve ρ = ρ MSE (δ) into two regions. Formal noise sensitivity is bounded throughout the region ρ < ρ MSE (δ) and is unbounded throughout the region ρ > ρ MSE (δ).The phase boundary ρ = ρ MSE (δ) is identical to the previously-known phase transition curve for equivalence of 1 − 0 minimization in the k-sparse noiseless case. Hence a single phase boundary describes the fundamental phase transitions both for the noiseless and noisy cases.Extensive computational experiments validate the predictions of this formalism, including the existence of game theoretical structures underlying it (saddlepoints in the payoff, least-favorable signals and maximin penalization).Underlying our formalism is an approximate message passing soft thresholding algorithm (AMP) introduced earlier by the authors. Other papers by the authors detail expressions for the formal MSE of AMP and its close connection to 1 -penalized reconstruction. Here we derive the minimax formal MSE of AMP and then read out results for 1 -penalized reconstruction.

High dimensional robust M-estimation: asymptotic variance via approximate message passing

“…The last section raises the possibility that the phenomena found in this paper for Mestimation in the p < n case are actually isomorphic to those found in our previous work on penalized regression in the p > n case; [5,9,11,13]. Here we merely content ourselves with sketching a few similarities.…”

“…In the classical setting no such effect is visible. One could say that the central fact about the high-dimensional setting revealed here as well as in our earlier work [5,9,11,13], is that when there are so many parameters to estimate, one cannot really insulate the estimation of any one parameter from the errors in estimation of all the other parameters. …”

“…While this treatment focuses on estimating a certain partition function, in the case of strongly convex ρ it should be possible to extract properties of the minimizer from a 'zero-temperature' limit. 13 However, as we pointed out at the end of Sect. 4, [15] proves that the case of general Gaussian designs X i ∼ N(0, /n) can be reduced to standard Gaussian designs X i ∼ N(0, I p× p /n) by a change of variables…”

“…That approach grew out of a series of earlier papers studying the compressed sensing problem [5,9,11,13]. From the perspective of this paper, those papers considered the same regression model (1) as here; however, they emphasized the challenging asymptotic regime where there are fewer observations than predictors, (i.e.…”

“…• The precise form of various terms in (13), (14) (15) is dictated by the statistical assumptions that we are making on the design X. In particular the memory terms are crucial for the state evolution analysis to hold.…”