Flexible Learning of Sparse Neural Networks via Constrained L0 Regularizations

Gallego-Posada, Jose; Rios, Juan Ramirez de los; Erraqabi, Akram

doi:10.52591/lxai202112071

Cited by 6 publications

(13 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ip i∈Ip l(w; x i , y i ), where I p is an index set of local data samples, I p := |I p | is the number of local data samples, I := P p=1 I p is the total number of data samples, l represents a loss function, and x i and y i represent data features and data labels, respectively. We note that most FL literature assumes W := R n , but more recently the importance of imposing hard constraints instead of soft constraints (i.e., penalizing in the objective function) has been discussed in the ML community [15,32].…”

Section: Federated Learningmentioning

confidence: 99%

“…Even though locally stored data are not shared with other agents, it is still possible to The aforementioned DP algorithms have been developed for unconstrained models; but, in general, constraints are necessary, for example, in most optimal control problems including distributed control of power flow. Also, imposing hard constraints on ML models (instead of penalizing constraints in the objective function as in [40,10]) is increasingly considered for purposes such as improving accuracy, explaining decisions suggested by ML models, promoting fairness, and observing some physical laws (e.g., [15,17,32,47,50]). This calls for developing DP algorithms suitable for the general constrained optimization model, like the form (1.1).…”

mentioning

confidence: 99%

See 1 more Smart Citation

Differentially Private Distributed Convex Optimization

Ryu¹,

Kim²

2023

Preprint

View full text Add to dashboard Cite

This paper considers distributed optimization (DO) where multiple agents cooperate to minimize a global objective function, expressed as a sum of local objective functions, subject to some constraints. In DO, each agent iteratively solves a local optimization model constructed by its own data and communicates some information (e.g., a local solution) with its neighbors in a communication network until a global solution is obtained. Even though locally stored data are not shared with other agents, it is still possible to reconstruct the data from the information communicated among agents, which could limit the practical usage of DO in applications with sensitive data. To address this issue, we propose a privacy-preserving DO algorithm for constrained convex optimization models, which provides a statistical guarantee of data privacy, known as differential privacy, and a sequence of iterates that converges to an optimal solution in expectation. The proposed algorithm generalizes a linearized alternating direction method of multipliers by introducing a multiple local updates technique to reduce communication costs and incorporating an objective perturbation method in the local optimization models to compute and communicate randomized feasible local solutions that cannot be utilized to reconstruct the local data, thus preserving data privacy. Under the existence of convex constraints, we show that, while both algorithms provide the same level of data privacy, the objective perturbation used in the proposed algorithm can provide better solutions than does the widely adopted output perturbation method that randomizes the local solutions by adding some noise. We present the details of privacy and convergence analyses and numerically demonstrate the effectiveness of the proposed algorithm by applying it in two different applications, namely, distributed control of power flow and federated learning, where data privacy is of concern.

show abstract

Section: Federated Learningmentioning

confidence: 99%

mentioning

confidence: 99%

Differentially Private Distributed Convex Optimization

Ryu¹,

Kim²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We follow [22,23,34] to formulate the structured pruning task as a regularized learning problem, which aims to learn a sparse model. Let f (•; θ) be a model with parameters θ = {θj } n j=1 , where each θj is a group of parameters (e.g., an attention head) and n is the number of groups.…”

Section: Structured Pruning Using L0 Regularizationmentioning

confidence: 99%

“…( 1) using gradient descent because the gates are discrete. Louizos et al [34] propose a reparameterization trick to make the loss differentiable, which has been widely used in sparse model learning. Here we only introduce their final approach.…”

Section: Structured Pruning Using L0 Regularizationmentioning

confidence: 99%

See 1 more Smart Citation

Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding

Peng¹,

Kim²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow. Model compression techniques such as pruning aim to reduce the model size and computation without degradation in accuracy. Prior studies focus on the pruning of Transformers; however, speech models not only utilize a stack of Transformer blocks, but also combine a frontend network based on multiple convolutional layers for low-level feature representation learning. This frontend has a small size but a heavy computational cost. In this work, we propose three task-specific structured pruning methods to deal with such heterogeneous networks. Experiments on LibriSpeech and SLURP show that the proposed method is more accurate than the original wav2vec2-base with 10% to 30% less computation, and is able to reduce the computation by 40% to 50% without any degradation.

show abstract

Phase-field Models for Lightweight Graph Convolutional Networks

Sahbi

2023

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

In this paper, we design lightweight graph convolutional networks (GCNs) using a particular class of regularizers, dubbed as phase-field models (PFMs). PFMs exhibit a bi-phase behavior using a particular ultra-local term that allows training both the topology and the weight parameters of GCNs as a part of a single "end-to-end" optimization problem. Our proposed solution also relies on a reparametrization that pushes the mask of the topology towards binary values leading to effective topology selection and high generalization while implementing any targeted pruning rate. Both masks and weights share the same set of latent variables and this further enhances the generalization power of the resulting lightweight GCNs. Extensive experiments conducted on the challenging task of skeleton-based recognition show the outperformance of PFMs against other staple regularizers as well as related lightweight design methods.

show abstract

Flexible Learning of Sparse Neural Networks via Constrained L0 Regularizations

Cited by 6 publications

References 0 publications

Differentially Private Distributed Convex Optimization

Differentially Private Distributed Convex Optimization

Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding

Phase-field Models for Lightweight Graph Convolutional Networks

Contact Info

Product

Resources

About