Quickly Finding the Best Linear Model in High Dimensions via Projected Gradient Descent

Abstract

We study the problem of finding the best linear model that can minimize least-squares loss given a dataset. While this problem is trivial in the low-dimensional regime, it becomes more interesting in high-dimensions where the population minimizer is assumed to lie on a manifold such as sparse vectors. We propose projected gradient descent (PGD) algorithm to estimate the population minimizer in the finite sample regime. We establish linear convergence rate and data-dependent estimation error bounds for PGD. Our contributions include: 1) The results are established for heavier tailed sub-exponential distributions besides sub-gaussian. 2) We directly analyze the empirical risk minimization and do not require a realizable model that connects input data and labels. 3) Our PGD algorithm is augmented to learn the bias terms which boosts the performance. The numerical experiments validate our theoretical results.

Generic error bounds for the generalized Lasso with sub-exponential data

“…To the best of our knowledge, Proposition 1.4 is a new result, but it bears resemblance with a recent finding of Sattar and Oymak [46,Thm. 3.4], who consider a similar model setup with sub-exponential input vectors.…”

“…A more astonishing feature of the generalized Lasso (LS K ) is its ability to deal with non-linear relations between x and y. In fact, inspired by a classical result of Brillinger [5], a recent work of Plan and Vershynin [42] shows that for Gaussian input vectors, (LS K ) yields a consistent estimator for single-index models, i.e., y = f ( x, β 0 ) with an unknown, non-linear distortion function f : R → R. This finding has triggered a lot of related and follow-up research, e.g., see [14,16,19,38,46,[52][53][54]. We note that these works form only a small fraction of a whole research area on non-linear observation models, lying at the interface of statistics, learning theory, signal processing, and compressed sensing.…”

“…The present work is inspired by the general framework developed in [15], which enables a theoretical analysis of (LS K ) for a large class of semi-parametric observation models (see also the technical report [17]). More specifically, we will not assume an explicit functional relationship between x and y, such as for single-index models; note that a similar viewpoint is taken by Sattar and Oymak [46], who refrain from a realizable model connecting the input and output. Adopting this abstract setup, we intend to address the following parameter estimation problem: Problem 1.…”

A unified analysis of convex and non-convex lp-ball projection problems

“…Here γ k > 0 is the step size at iteration k. Bahmani and Raj (Bahmani and Raj, 2013) analyzed the rate of convergence of x k to s as a function of p, showing that the sufficient conditions for exact signal recovery become more stringent while robustness to noise and convergence rate worsen, as p increases from 0 to 1. Oymak et al (Oymak et al, 2017;Sattar and Oymak, 2020) extended the analysis for more general (non-convex) constraint sets including p -balls. Their experiments compared cases p = 0, 0.5, and 1, and suggest that while p = 0 outperforms p = 1, it is dominated by p = 0.5.…”

Unfolded Algorithms for Deep Phase Retrieval

“…Specifically, first-order methods are among the most popular and well-established iterative optimization techniques due to their low per-iteration complexity and efficiency in complex scenarios. One of the most prominent first-order optimization techniques suitable for our problem in ( 14) is the projected gradient descent algorithm (PGD) [54]- [59]. In particular, assuming z 0 is the initial point for the algorithm, the l-th iteration of PGD to approximate the solution to ( 14) can be defined as follows:…”