Compressing Deep Networks by Neuron Agglomerative Clustering

Wang, Lina; Liu, Wenxue; Liu, Xiang; Zhong, Guoqiang; Roy, Partha Pratim; Dong, Junyu; Huang, Kaizhu

doi:10.3390/s20216033

Cited by 4 publications

(2 citation statements)

References 32 publications

(31 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach takes into account that several connections may share the same weight value, and then fine-tunes those shared weights. In the case of feedforward structures, this strategy was already successfully employed to minimize the complexity of NN models [46], [69]- [71]. In this paper, we use the same method as in [46], but modify it for the recurrent layers as well.…”

Section: B Weights Clusteringmentioning

confidence: 99%

Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation

Freire

Napoli

Costa

et al. 2023

J. Lightwave Technol.

View full text Add to dashboard Cite

In this paper, a new methodology is proposed that allows for the low-complexity development of neural network (NN) based equalizers for the mitigation of impairments in highspeed coherent optical transmission systems. In this work, we provide a comprehensive description and comparison of various deep model compression approaches that have been applied to feed-forward and recurrent NN designs. Additionally, we evaluate the influence these strategies have on the performance of each NN equalizer. Quantization, weight clustering, pruning, and other cutting-edge strategies for model compression are taken into consideration. In this work, we propose and evaluate a Bayesian optimization-assisted compression, in which the hyperparameters of the compression are chosen to simultaneously reduce complexity and improve performance. Next, this paper presents four distinct metrics (RMpS, BoP, NABS, and NLGs) that are discussed here that can be used to evaluate the amount of computing complexity required by various compression algorithms. These measurements can serve as a benchmark for evaluating the relative effectiveness of various NN equalizers when compression approaches are used. In conclusion, the trade-off between the complexity of each compression approach and its performance is evaluated by utilizing both simulated and experimental data in order to complete the analysis. By utilizing optimal compression approaches, we show that it is possible to design an NN-based equalizer that is simpler to implement and has better performance than the conventional digital back-propagation (DBP) equalizer with only one step per span. This is accomplished by reducing the number of multipliers used in the NN equalizer after applying the weighted clustering and pruning algorithms. Furthermore, we demonstrate that an equalizer based on NN can also achieve superior performance while still maintaining the same degree of complexity as the full electronic chromatic dispersion compensation block. We conclude our analysis by highlighting open questions and existing challenges, as well as possible future research directions.

show abstract

Section: B Weights Clusteringmentioning

confidence: 99%

Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation

Freire

Napoli

Costa

et al. 2023

J. Lightwave Technol.

View full text Add to dashboard Cite

show abstract

“…However, the existing research work does not consider the heterogeneous capabilities of IoT devices, dynamic changes of environmental conditions, and is difficult to achieve real-time adaptive decision-making under the diversified environment configuration and high computational complexity of problem solving. It is worth noting that, the above work is orthogonal to the compression and acceleration methods that use weight pruning [25,26], quantization [27,28] and low-precision inference [29,30] to reduce the computational cost of DNN models. At the same time, these two technologies are used to accelerate the DNN inference.…”

Section: Introductionmentioning

confidence: 99%

Machine Learning-based Optimal Framework for Internet of Things Networks

Alsafasfeh¹,

Arida²,

Saraereh³

2022

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Deep neural networks (DNN) are widely employed in a wide range of intelligent applications, including image and video recognition. However, due to the enormous amount of computations required by DNN. Therefore, performing DNN inference tasks locally is problematic for resourceconstrained Internet of Things (IoT) devices. Existing cloud approaches are sensitive to problems like erratic communication delays and unreliable remote server performance. The utilization of IoT device collaboration to create distributed and scalable DNN task inference is a very promising strategy. The existing research, on the other hand, exclusively looks at the static split method in the scenario of homogeneous IoT devices. As a result, there is a pressing need to investigate how to divide DNN tasks adaptively among IoT devices with varying capabilities and resource constraints, and execute the task inference cooperatively. Two major obstacles confront the aforementioned research problems: 1) In a heterogeneous dynamic multi-device environment, it is difficult to estimate the multi-layer inference delay of DNN tasks; 2) It is difficult to intelligently adapt the collaborative inference approach in real time. As a result, a multi-layer delay prediction model with fine-grained interpretability is proposed initially. Furthermore, for DNN inference tasks, evolutionary reinforcement learning (ERL) is employed to adaptively discover the approximate best split strategy. Experiments show that, in a heterogeneous dynamic environment, the proposed framework can provide considerable DNN inference acceleration. When the number of devices is 2, 3, and 4, the delay acceleration of the proposed algorithm is 1.81 times, 1.98 times and 5.28 times that of the EE algorithm, respectively.

show abstract