Learning sparse deep neural networks using efficient structured projections on convex constraints for green AI

Abstract

Deep neural networks (DNN) have been applied recently to different domains and perform better than classical state-of-the-art methods. However the high level of performances of DNNs is most often obtained with networks containing millions of parameters and for which training requires substantial computational power. To deal with this computational issue proximal regularization methods have been proposed in the literature but they are time consuming. In this paper, we propose instead a constrained approach. We provide the general framework for this new projection gradient method. Our algorithm iterates a gradient step and a projection on convex constraints. We studied algorithms for different constraints: the classical 1 unstructured constraint and structured constraints such as the 2,1 constraint (Group LASSO). We propose a new 1,1 structured constraint for which we provide a new projection algorithm. Finally, we used the recent "Lottery optimizer" replacing the threshold by our 1,1 projection. We demonstrate the effectiveness of this method with three popular datasets (MNIST, Fashion MNIST and CIFAR). Experiments with these datasets show that our projection method using this new 1,1 structured constraint provides the best decrease in memory and computational power.

Learning a confidence score and the latent space of a new supervised autoencoder for diagnosis and prognosis in clinical metabolomic studies

“…Total Loss Following the work developed by [34], which proposed a double descent algorithm, we replaced the thresholding by our ℓ 1,1 projection and devised a new double descent algorithm (See Barlaud and Guyard [35]) as follows :…”

Detecting subtle transcriptomic perturbations induced by lncRNAs Knock-Down in single-cell CRISPRi screening using a new sparse supervised autoencoder neural network

“…The goal is to compute the network weights W minimizing the total loss which includes both the classification loss and the reconstruction loss. To perform feature selection, as large datasets often present a relatively small number of informative features, we also want to sparsify the network, following the work proposed in [34]. Thus, instead of the classical computationally expensive lagrangian regularization approach [35], we propose to minimize the following constrained approach [36]: We use the Cross Entropy (CE) Loss for the classification loss ℋ.…”

“…The goal is to compute the network weights W minimizing the total loss which includes both the classification loss and the reconstruction loss. To perform feature selection, as large datasets often present a relatively small number of informative features, we also want to sparsify the network, following the work proposed in [34].…”

Low-cost Thyroid Nodule Diagnosis Using a Supervised Autoencoder Neural network

“…We compute the ℓ 1,1 constraint with the following algorithm: we first compute the radius t i and then project the rows using the ℓ 1 adaptive constraint t i [56]. We use a double descent algorithm with ℓ 1,1 projection [44] as follows:…”