Hybrid Network Compression via Meta-Learning

In this paper, a new methodology is proposed that allows for the low-complexity development of neural network (NN) based equalizers for the mitigation of impairments in highspeed coherent optical transmission systems. In this work, we provide a comprehensive description and comparison of various deep model compression approaches that have been applied to feed-forward and recurrent NN designs. Additionally, we evaluate the influence these strategies have on the performance of each NN equalizer. Quantization, weight clustering, pruning, and other cutting-edge strategies for model compression are taken into consideration. In this work, we propose and evaluate a Bayesian optimization-assisted compression, in which the hyperparameters of the compression are chosen to simultaneously reduce complexity and improve performance. Next, this paper presents four distinct metrics (RMpS, BoP, NABS, and NLGs) that are discussed here that can be used to evaluate the amount of computing complexity required by various compression algorithms. These measurements can serve as a benchmark for evaluating the relative effectiveness of various NN equalizers when compression approaches are used. In conclusion, the trade-off between the complexity of each compression approach and its performance is evaluated by utilizing both simulated and experimental data in order to complete the analysis. By utilizing optimal compression approaches, we show that it is possible to design an NN-based equalizer that is simpler to implement and has better performance than the conventional digital back-propagation (DBP) equalizer with only one step per span. This is accomplished by reducing the number of multipliers used in the NN equalizer after applying the weighted clustering and pruning algorithms. Furthermore, we demonstrate that an equalizer based on NN can also achieve superior performance while still maintaining the same degree of complexity as the full electronic chromatic dispersion compensation block. We conclude our analysis by highlighting open questions and existing challenges, as well as possible future research directions.

show abstract

“…In Ref. [123], the authors have jointly considered network pruning and quantization in an end-to-end meta-learning framework. Other papers, e.g.…”

Section: Conclusion Open Problems and Research Directionsmentioning

confidence: 99%

Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation

Freire

Napoli

Costa

et al. 2023

J. Lightwave Technol.

View full text Add to dashboard Cite

show abstract

“…By consolidating both meta-learning and model compression, existing researches learn light-weight models that can quickly adapt to edge environments [263]- [266]. [264] proposes an end-to-end framework to seek layer-wise compression with meta-learning. [265] learns a meta-teacher that is generalizable across domains and guides the student model to solve domain-specific tasks.…”

Section: Meta-learningmentioning

confidence: 99%

Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI

Yao¹,

Zhang²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Influenced by the great success of deep learning via cloud computing and the rapid development of edge chips, research in artificial intelligence (AI) has shifted to both of the computing paradigms, i.e., cloud computing and edge computing. In recent years, we have witnessed significant progress in developing more advanced AI models on cloud servers that surpass traditional deep learning models owing to model innovations (e.g., Transformers, Pretrained families), explosion of training data and soaring computing capabilities. However, edge computing, especially edge and cloud collaborative computing, are still in its infancy to announce their success due to the resource-constrained IoT scenarios with very limited algorithms deployed. In this survey, we conduct a systematic review for both cloud and edge AI. Specifically, we are the first to set up the collaborative learning mechanism for cloud and edge modeling with a thorough review of the architectures that enable such mechanism. We also discuss potentials and practical experiences of some on-going advanced edge AI topics including pretraining models, graph neural networks and reinforcement learning. Finally, we discuss the promising directions and challenges in this field.

show abstract

“…Through parameter quantization, the resolution of the parameters can be reduced to 16-bit, 8-bit, 4-bit, and even 1-bit with little loss of the accuracy in some tasks. [26][27][28] Prakosa et al 29 adopted the K-D method to improve the performance of the pruned network. Blakeney et al 30 proposed a parallel block-wise K-D method to compress the deep neural networks.…”

Section: Introductionmentioning

confidence: 99%

Iterative-AMC: a novel model compression and structure optimization method in mechanical system safety monitoring

Peng

et al. 2023

Structural Health Monitoring

View full text Add to dashboard Cite

With the rapid development of artificial intelligence, various fault diagnosis methods based on the deep neural networks have made great advances in mechanical system safety monitoring. To get the high accuracy for the fault diagnosis, researchers tend to adopt the deep network layers and amount of neurons or kernels in each layer. This results in a large redundancy and the structure uncertainty of the fault diagnosis networks. Moreover, it is hard to deploy these networks on the embedded platforms because of the large scales of the network parameters. This brings huge challenges to the practical application of the intelligent diagnosis algorithms. To solve the above problems, an iterative automatic machine compression method, named Iterative-AMC, is proposed in this paper. The proposed method aims to automatically compress and optimize the structure of the large-scale neural networks. Experiments are carried out based on two test benches. With the proposed Iterative-AMC method, the problems of the parameter redundancy and the structure uncertainty can be addressed. The scale of the original network can be greatly compressed, and the compressed fault diagnosis network is successfully deployed on a small-scale FPGA chip.

show abstract

Hybrid Network Compression via Meta-Learning

Cited by 5 publications

References 25 publications

Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation

Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation

Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI

Iterative-AMC: a novel model compression and structure optimization method in mechanical system safety monitoring

Contact Info

Product

Resources

About