Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning

Abstract

As deep neural networks are growing in size and being increasingly deployed to more resource-limited devices, there has been a recent surge of interest in network pruning methods, which aim to remove less important weights or activations of a given network. A common limitation of most existing pruning techniques, is that they require pre-training of the network at least once before pruning, and thus we can benefit from reduction in memory and computation only at the inference time. However, reducing the training cost of neural networks with rapid structural pruning may be beneficial either to minimize monetary cost with cloud computing or to enable on-device learning on a resource-limited device. Recently introduced random-weight pruning approaches can eliminate the needs of pretraining, but they often obtain suboptimal performance over conventional pruning techniques and also does not allow for faster training since they perform unstructured pruning. To overcome their limitations, we propose Set-based Task-Adaptive Meta Pruning (STAMP), which task-adaptively prunes a network pretrained on a large reference dataset by generating a pruning mask on it as a function of the target dataset. To ensure maximum performance improvements on the target task, we meta-learn the mask generator over different subsets of the reference dataset, such that it can generalize well to any unseen datasets within a few gradient steps of training. We validate STAMP against recent advanced pruning methods on benchmark datasets, on which it not only obtains significantly improved compression rates over the baselines at similar accuracy, but also orders of magnitude faster training speed.Preprint. Under review.

MetaGater: Fast Learning of Conditional Channel Gated Networks via Federated Meta-Learning

“…Note that an independent and concurrent work [56] also proposes to utilize meta-learning for rapid structural pruning of neural networks. We highlight the main differences below: 1) [56] relies on a centralized meta-learning method where the nodes are required to submit data to a central platform, whereas we consider a more realistic distributed setup and propose a new federated meta-learning approach to fit the specific efficiency problem in our work. 2) [56] takes a stochastic approach and learns a task-specific Bernoulli distribution for mask generation, which however could possibly generate masks that lead to significant performance degradation.…”

“…We highlight the main differences below: 1) [56] relies on a centralized meta-learning method where the nodes are required to submit data to a central platform, whereas we consider a more realistic distributed setup and propose a new federated meta-learning approach to fit the specific efficiency problem in our work. 2) [56] takes a stochastic approach and learns a task-specific Bernoulli distribution for mask generation, which however could possibly generate masks that lead to significant performance degradation. In stark contrast, we develop a deterministic approach by learning a task-specific channel gating module, and also provide theoretic foundations by carrying out a thorough convergence analysis of the proposed federated meta-learning algorithm.…”