Dynamic Screening: Accelerating First-Order Algorithms for the Lasso and Group-Lasso

Abstract

International audienceRecent computational strategies based on screening tests have been proposed to accelerate algorithms addressing penalized sparse regression problems such as the Lasso. Such approaches build upon the idea that it is worth dedicating some small computational effort to locate inactive atoms and remove them from the dictionary in a preprocessing stage so that the regression algorithm working with a smaller dictionary will then converge faster to the solution of the initial problem. We believe that there is an even more efficient way to screen the dictionary and obtain a greater acceleration: inside each iteration of the regression algorithm, one may take advantage of the algorithm computations to obtain a new screening test for free with increasing screening effects along the iterations. The dictionary is henceforth dynamically screened instead of being screened statically, once and for all, before the first iteration. We formalize this dynamic screening principle in a general algorithmic scheme and apply it by embedding inside a number of first-order algorithms adapted existing screening tests to solve the Lasso or new screening tests to solve the Group-Lasso. Computational gains are assessed in a large set of experiments on synthetic data as well as real-world sounds and images. They show both the screening efficiency and the gain in terms running times

Joint Screening Tests for Lasso

“…We find in the literature different definitions for the center c and the radius 1 − τ (see [5][6][7][8][9][10][11]) leading to screening tests of different effectiveness. It is easy to see that the implementation of the standard screening test (8) necessitates the evaluation of max θ∈R a i , θ for each atom of the dictionary.…”

“…Therefore, the conception of computationally-efficient techniques to solve (1) has become an active field of research [2][3][4]. One important contribution in this field is the so-called "safe" 2 screening technique proposed by El Ghaoui et al in [5] and refined in several subsequent works [6][7][8][9][10][11][12][13]. These methodologies aim at decreasing the dimensionality of the problem to handle by identifying some of the zeros of the target solution x λ via simple tests.…”

Stable Safe Screening and Structured Dictionaries for Faster $\ell _{1}$ Regularization

“…To obtain dynamic spherical regions, we need dual feasible points that change over iterations (θ F = θ t at iteration t) and leverage the current state of the optimization algorithm. Following [21], this can be achieved by defining θ t proportional to the current residual ρ t := y − Ax t , since the dual solution θ ⋆ is proportional to y − Ax ⋆ . Taking z = ρ t and…”

“…2) Dynamic Safe sphere [21]: With the dual feasible point defined in (11), (9) yields R = B(c, R) with…”

“…This region is provably safe for any dual feasible point θ t ∈ ∆ A [22, Theorem 2]. The authors, following [21], propose to use θ t defined by (11). Note that the cited safe regions reuse quantities computed along the iterations of classical proximal algorithms such as the residual ρ t , the inner products A T ρ t , or the duality gap (which typically serves as a stopping criterion).…”