Orthant Based Proximal Stochastic Gradient Method for $\ell_1$-Regularized Optimization

Abstract

Sparsity-inducing regularization problems are ubiquitous in machine learning applications, ranging from feature selection to model compression. In this paper, we present a novel stochastic method -Orthant Based Proximal Stochastic Gradient Method (OBProx-SG) -to solve perhaps the most popular instance, i.e., the 1-regularized problem. The OBProx-SG method contains two steps: (i) a proximal stochastic gradient step to predict a support cover of the solution; and (ii) an orthant step to aggressively enhance the sparsity level via orthant face projection. Compared to the state-of-the-art methods, e.g., Prox-SG, RDA and Prox-SVRG, the OBProx-SG not only converges to the global optimal solutions (in convex scenario) or the stationary points (in non-convex scenario), but also promotes the sparsity of the solutions substantially. Particularly, on a large number of convex problems, OBProx-SG outperforms the existing methods comprehensively in the aspect of sparsity exploration and objective values. Moreover, the experiments on non-convex deep neural networks, e.g., MobileNetV1 and ResNet18, further demonstrate its superiority by achieving the solutions of much higher sparsity without sacrificing generalization accuracy.

CDFI: Compression-Driven Network Design for Frame Interpolation

“…To the best of our knowledge, we are the first to explore the over-parameterization issue appearing in the state-of-the-art DNN models for video interpolation. Concretely, we compress the recently proposed AdaCoF [32] via fine-grained pruning [67] based on sparsity-inducing optimization [7], and show that a 10× compressed AdaCoF is still able to maintain a similar benchmark performance as before, indicating a considerable amount of redundancy in the original model. The compression provides us two direct benefits: (i) it helps us understand the model architecture in depth, which in turn inspires an efficient design; (ii) the obtained compact model makes more room for further improvements that could potentially boost the performance to a new level.…”

“…It is known that with appropriately chosen λ the formulation (2) promotes a sparse solution, with which one can easily identify those important connections among neurons, namely the ones corresponding to non-zero weights. Towards solving (2), we utilize the newly proposed orthant-based stochastic method [7] for its efficient mechanism in promoting sparsity and less performance regression compared with other solvers. By solving the 1regularized problem (2), we indeed perform a fine-grained pruning since zeros are promoted in an unstructured manner.…”

“…We use the pretrained model provided by the authors. Starting with the 1regularized problem (2), where λ is set as 10 −4 , we run the orthant-based stochastic solver [7] for 20 epochs by feeding the model with only 1000 video triplets from Vimeo-90K [62]. For each epoch, we record the network density and the PSNR evaluated on the Middlebury dataset [1], as plotted in Figure 5.…”

Half-Space Proximal Stochastic Gradient Method for Group-Sparsity Regularized Problem

“…Although Prox-SVRG/SAGA may have better theoretical convergence property than Prox-SG, they require higher time and space complexity to compute or estimate full gradient on a huge mini-batch or store previous gradient, which may be prohibitive for large-scale training especially when the memory is often limited. Besides, it is well noticed that SVRG does not work as desired on the popular non-convex deep learning applications (Defazio & Bottou, 2019;Chen et al, 2020). In contrast, Prox-SG is efficient and can also achieves the good initialization assumption in Theorem 1.…”

“…Unless Ω(x) is x 1 where each g ∈ G is singleton, then S k becomes an orthant face(Chen et al, 2020).…”