Structured least squares with bounded data uncertainties

We propose novel randomized optimization methods for high-dimensional convex problems based on restrictions of variables to random subspaces. We consider oblivious and data-adaptive subspaces and study their approximation properties via convex duality and Fenchel conjugates. A suitable adaptive subspace can be generated by sampling a correlated random matrix whose second order statistics mirror the input data. We illustrate that the adaptive strategy can significantly outperform the standard oblivious sampling method, which is widely used in the recent literature. We show that the relative error of the randomized approximations can be tightly characterized in terms of the spectrum of the data matrix and Gaussian width of the dual tangent cone at optimum. We develop lower bounds for both optimization and statistical error measures based on concentration of measure and Fano's inequality. We then present the consequences of our theory with data matrices of varying spectral decay profiles. Experimental results show that the proposed approach enables significant speed ups in a wide variety of machine learning and optimization problems including logistic regression, kernel classification with random convolution layers and shallow neural networks with rectified linear units.

show abstract

“…Plugging-in this inequality into (71) as well as the strong convexity inequality ∆, g y * t − g z * 1 µ ∆ 2 2 , we obtain that…”

Section: E Proof Of Theoremmentioning

confidence: 99%

Adaptive and Oblivious Randomized Subspace Methods for High-Dimensional Optimization: Sharp Analysis and Lower Bounds

Lacotte

Pilancı

2022

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…If x is fixed then there exists many algorithms to solve for a sparse p [12]. Therefore a local optimum can be found using an alternating minimizations algorithm [13] where we chose Orthogonal Matching Pursuit (OMP) [14] in the intermediate step for its simplicity:…”

Section: Iii-a Alternating Minimizations Algorithm To Solve P0mentioning

confidence: 99%

Recovery of sparse perturbations in Least Squares problems

Pilancı

Arıkan

2011

2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We show that the exact recovery of sparse perturbations on the coefficient matrix in overdetermined Least Squares problems is possible for a large class of perturbation structures. The well established theory of Compressed Sensing enables us to prove that if the perturbation structure is sufficiently incoherent, then exact or stable recovery can be achieved using linear programming. We derive sufficiency conditions for both exact and stable recovery using known results of 0/ 1 equivalence. However the problem turns out to be more complicated than the usual setting used in various sparse reconstruction problems. We propose and solve an optimization criterion and its convex relaxation to recover the perturbation and the solution to the Least Squares problem simultaneously. Then we demonstrate with numerical examples that the proposed method is able to recover the perturbation and the unknown exactly with high probability. The performance of the proposed technique is compared in blind identification of sparse multipath channels.

show abstract

“…In [24], it is shown that for high SNR the covariance matrix of the STLS estimator can be approximated by (5) If has a large condition number, the variance can be extremely large. It is usually noted in applications that at low SNR, the error variance is even larger than its approximation in (5) [25], [26].…”

Section: B Regularized-structured Total Least Squares Approachmentioning

confidence: 99%

“…Consider the single parameter equation below: (24) The corresponding structures are (25) Define the cost of given by (26) which corresponds to a constant multiple of the negative loglikelihood given for the observation where is a zero-mean Gaussian random variable. Fig.…”

Section: Analysis Of Estimator Performance In An Illustrative Examentioning

confidence: 99%

Structured Least Squares Problems and Robust Estimators

Pilancı

Arıkan

Pınar

2010

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

Abstract-A novel approach is proposed to provide robust and accurate estimates for linear regression problems when both the measurement vector and the coefficient matrix are structured and subject to errors or uncertainty. A new analytic formulation is developed in terms of the gradient flow of the residual norm to analyze and provide estimates to the regression. The presented analysis enables us to establish theoretical performance guarantees to compare with existing methods and also offers a criterion to choose the regularization parameter autonomously. Theoretical results and simulations in applications such as blind identification, multiple frequency estimation and deconvolution show that the proposed technique outperforms alternative methods in mean-squared error for a significant range of signal-to-noise ratio values.

show abstract

Structured least squares with bounded data uncertainties

Cited by 6 publications

References 6 publications

Adaptive and Oblivious Randomized Subspace Methods for High-Dimensional Optimization: Sharp Analysis and Lower Bounds

Adaptive and Oblivious Randomized Subspace Methods for High-Dimensional Optimization: Sharp Analysis and Lower Bounds

Recovery of sparse perturbations in Least Squares problems

Structured Least Squares Problems and Robust Estimators

Contact Info

Product

Resources

About