Implicit Regularization for Optimal Sparse Recovery

Abstract

We investigate implicit regularization schemes for gradient descent methods applied to unpenalized least squares regression to solve the problem of reconstructing a sparse signal from an underdetermined system of linear measurements under the restricted isometry assumption. For a given parametrization yielding a non-convex optimization problem, we show that prescribed choices of initialization, step size and stopping time yield a statistically and computationally optimal algorithm that achieves the minimax rate with the same cost required to read the data up to poly-logarithmic factors. Beyond minimax optimality, we show that our algorithm adapts to instance difficulty and yields a dimension-independent rate when the signal-to-noise ratio is high enough. Key to the computational efficiency of our method is an increasing step size scheme that adapts to refined estimates of the true solution. We validate our findings with numerical experiments and compare our algorithm against explicit 1 penalization. Going from hard instances to easy ones, our algorithm is seen to undergo a phase transition, eventually matching least squares with an oracle knowledge of the true support.

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

“…The linear diagonal neural networks we consider have been studied in the case of gradient descent [33] and stochastic gradient descent with label noise [15]. In both cases the authors show that this model has the ability to implicitly bias the training procedure to help retrieve a sparse predictor.…”

“…For the sake of completeness, the study of diagonal linear networks of arbitrary depth p ≥ 3 is done in Appendix E.2. Also note that additionally to being a toy neural model, it has received recent attention on its own for its practical ability to induce sparsity [33,34,15] or to solve phase retrieval problems [37].…”

Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisation

“…The quadratic parametrisations which we consider have become popular lately (Vaškevičius et al, 2019) since, despite their simplicity, they already enable to grasp the complexity of more general networks. Indeed, they highlight important aspects of the theoretical concerns of modern machine learning: the neural tangent kernel regime, the roles of overparametrisation and of the initialisation (Woodworth et al, 2020).…”