&lt;title&gt;Examples of basis pursuit&lt;/title&gt;

Chen, Scott; Donoho, David L.

doi:10.1117/12.217610

Cited by 34 publications

(34 citation statements)

References 1 publication

(3 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Viceversa, any minimizer θ * of the problem (2) corresponds to one (or more) AMP fixed points of the form ( θ * , R * , b * ). 7 Here and below, given f : 8 This equation always admits at least one solution since b → (r ; b) is continuous in b ≥ 0, with (r ; 0) = 0 and (for ρ strictly convex) (r ; ∞) = 1, cf. Proposition 6.3.…”

Section: Relation To M-estimationmentioning

confidence: 99%

“…n/ p(n) → δ ∈ (0, 1)) so that even in the noiseless case, the equations Y = Xθ would be underdetermined. In the p > n setting, it became popular to use 1 -penalized least squares (Lasso [7,34]). That series of papers considered the Lasso convex optimization problem in the case of X with iid N(0, 1/n) entries (just as here) and followed the same 3-step strategy we use here; namely, (1) Introducing an AMP algorithm; (2) Obtaining the asymptotic distribution of AMP by state evolution; and (3) Showing that AMP agrees with the Lasso solution in the large-n limit.…”

Section: Underlying Toolsmentioning

confidence: 99%

See 1 more Smart Citation

High dimensional robust M-estimation: asymptotic variance via approximate message passing

Donoho

Montanari

2015

Probab. Theory Relat. Fields

155

195

View full text Add to dashboard Cite

In a recent article, El Karoui et al. (Proc Natl Acad Sci 110(36):14557-14562, 2013) study the distribution of robust regression estimators in the regime in which the number of parameters p is of the same order as the number of samples n. Using numerical simulations and 'highly plausible' heuristic arguments, they unveil a striking new phenomenon. Namely, the regression coefficients contain an extra Gaussian noise component that is not explained by classical concepts such as the Fisher information matrix. We show here that that this phenomenon can be characterized rigorously using techniques that were developed by the authors for analyzing the Lasso estimator under high-dimensional asymptotics. We introduce an approximate message passing (AMP) algorithm to compute M-estimators and deploy state evolution to evaluate the operating characteristics of AMP and so also M-estimates. Our analysis clarifies that the 'extra Gaussian noise' encountered in this problem is fundamentally similar to phenomena already studied for regularized least squares in the setting n < p. Mathematics Subject Classification 62F10 · 62F12 · 60F99 M-estimation under high dimensional asymptoticsConsider the traditional linear regression modelB Andrea Montanari 1 We are interested in estimating θ 0 from observed data 2 (Y, X) using a traditional M-estimator, defined by a non-negative convex function ρ : R → R ≥0 :where u, v = m i=1 u i v i is the standard scalar product in R m , and θ is chosen arbitrarily if there is multiple minimizers.Although this is a completely traditional problem, we consider it under highdimensional asymptotics where the number of parameters p and the number of observations n are both tending to infinity, at the same rate. This is becoming a popular asymptotic model owing to the modern awareness of 'big data' and 'data deluge'; but also because it leads to entirely new phenomena. Extra Gaussian noise due to high-dimensional asymptoticsClassical statistical theory considered the situation where the number of regression parameters p is fixed and the number of samples n is tending to infinity. The asymptotic distribution was found by Huber [2,18] to be normal N(0, V) where the asymptotic variance matrix V is given byhere ψ = ρ is the score function of the M-estimator and V (ψ, F) = ( ψ 2 dF)/ ( ψ dF) 2 the asymptotic variance functional of [17], and (X T X) the usual Gram matrix associated with the least-squares problem. Importantly, it was found that for efficient estimation-i.e. the smallest possible asymptotic variance-the optimal M-estimator depended on the probability distribution F W of the errors W . Choosing ψ(x) = (log f W (x)) (with f W the density of W ), the asymptotic variance functional yields V (ψ, F W ) = 1/I (F W ), with I (F) denoting the Fisher information. This achieves the fundamental limit on the accuracy of M-estimators [18]. In modern statistical practice there is increasing interest in applications where the number of explanatory variables p is very large, and comparable to n. Examples of...

show abstract

Section: Relation To M-estimationmentioning

confidence: 99%

Section: Underlying Toolsmentioning

confidence: 99%

High dimensional robust M-estimation: asymptotic variance via approximate message passing

Donoho

Montanari

2015

Probab. Theory Relat. Fields

155

195

View full text Add to dashboard Cite

show abstract

“…Both y and A are known, both x 0 and z 0 are unknown, and we seek an approximation to x 0 . A very popular approach estimates x 0 via the solution x 1,λ of the following convex optimization problem (P 2,λ,1 ) minimize 1 2 y − Ax Thousands of articles use or study this approach, which has variously been called LASSO, Basis Pursuit, or more prosaically, 1 -penalized least-squares [Tib96, CD95,CDS98]. There is a clear need to understand the extent to which (P 2,λ,1 ) accurately recovers x 0 .…”

Section: Introductionmentioning

confidence: 99%

The Noise-Sensitivity Phase Transition in Compressed Sensing

Donoho

Maleki

Montanari

2011

IEEE Trans. Inform. Theory

Self Cite

238

302

View full text Add to dashboard Cite

Consider the noisy underdetermined system of linear equations: y = Ax 0 + z 0 , with n × N measurement matrix A, n < N , and Gaussian white noise z 0 ∼ N(0, σ 2 I). Both y and A are known, both x 0 and z 0 are unknown, and we seek an approximation to x 0 . When x 0 has few nonzeros, useful approximations are often obtained by 1 -penalized 2 minimization, in which the reconstructionx 1,λ solves min y − Ax 2 2 /2 + λ x 1 . Evaluate performance by mean-squared error (MSE = E||xConsider matrices A with iid Gaussian entries and a large-system limit in which n, N → ∞ with n/N → δ and k/n → ρ. Call the ratio MSE/σ 2 the noise sensitivity. We develop formal expressions for the MSE ofx 1,λ , and evaluate its worst-case formal noise sensitivity over all types of k-sparse signals. The phase space 0 ≤ δ, ρ ≤ 1 is partitioned by curve ρ = ρ MSE (δ) into two regions. Formal noise sensitivity is bounded throughout the region ρ < ρ MSE (δ) and is unbounded throughout the region ρ > ρ MSE (δ).The phase boundary ρ = ρ MSE (δ) is identical to the previously-known phase transition curve for equivalence of 1 − 0 minimization in the k-sparse noiseless case. Hence a single phase boundary describes the fundamental phase transitions both for the noiseless and noisy cases.Extensive computational experiments validate the predictions of this formalism, including the existence of game theoretical structures underlying it (saddlepoints in the payoff, least-favorable signals and maximin penalization).Underlying our formalism is an approximate message passing soft thresholding algorithm (AMP) introduced earlier by the authors. Other papers by the authors detail expressions for the formal MSE of AMP and its close connection to 1 -penalized reconstruction. Here we derive the minimax formal MSE of AMP and then read out results for 1 -penalized reconstruction.

show abstract

“…Hence we can obtain the minimizer as a function ofỹ i by adding θ i to the graph i figure (19) and flipping the axis. The result is plotted in the next figure.…”

Section: 2mentioning

confidence: 99%

“…The LASSO (Least Absolute Shrinkage and Selection Operator) presented in [18], also known as Basis Pursuit DeNoising (BPDN) [19,20] is arguably the most successful method for sparse regression. The LASSO estimator is defined in terms of an optimization problem…”

Section: 2mentioning

confidence: 99%

Statistical estimation: from denoising to sparse regression and hidden cliques

Tramel¹,

Kumar²,

Giurgiu³

et al. 2015

Statistical Physics, Optimization, Inference, and Message-Passing Algorithms

View full text Add to dashboard Cite

Abstract. These notes review six lectures given by Prof. Andrea Montanari on the topic of statistical estimation for linear models. The first two lectures cover the principles of signal recovery from linear measurements in terms of minimax risk. Subsequent lectures demonstrate the application of these principles to several practical problems in science and engineering. Specifically, these topics include denoising of error-laden signals, recovery of compressively sensed signals, reconstruction of low-rank matrices, and also the discovery of hidden cliques within large networks.These are notes from the lecture of Andrea Montanari given at the autumn school "Statistical Physics, Optimization, Inference, and Message-Passing Algorithms", that took place in Les Houches,

show abstract

<title>Examples of basis pursuit</title>

Cited by 34 publications

References 1 publication

High dimensional robust M-estimation: asymptotic variance via approximate message passing

High dimensional robust M-estimation: asymptotic variance via approximate message passing

The Noise-Sensitivity Phase Transition in Compressed Sensing

Statistical estimation: from denoising to sparse regression and hidden cliques

Contact Info

Product

Resources

About