Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity

Abstract

Modern convolutional neural networks (CNNs) play a crucial role in computer vision applications. The intricacy of the application scenarios and the growing dataset both significantly raise the complexity of CNNs. As a result, they are often overparameterized and have significant computational costs. One potential solution for optimizing and compressing the CNNs is to replace convolutional layers with low-rank tensor decomposition. The most suitable technique for this is Canonical Polyadic (CP) decomposition. However, there are two primary issues with CP decomposition that lead to a significant loss in accuracy. Firstly, the selection of tensor ranks for CP decomposition is an unsolved issue. Secondly, degeneracy and instability are common problems in the CP decomposition of contractional tensors, which makes fine-tuning the compressed model difficult. In this study, a novel approach was proposed for compressing CNNs by using CP decomposition. The first step involves using the sensitivity of convolutional layers to determine the tensor ranks for CP decomposition effectively. Subsequently, to address the degeneracy issue and enhance the stability of the CP decomposition, two novel techniques were incorporated: optimization with sensitivity constraints and iterative fine-tuning based on sensitivity order. Finally, the proposed method was examined on common CNN structures for image classification tasks and demonstrated that it provides stable performance and significantly fewer reductions in classification accuracy.

Impulsive Control Discrete Fractional Neural Networks in Product Form Design: Practical Mittag-Leffler Stability Criteria

“…Hence, stability is one of the most important issues if an ANN model is to be used in practical applications [49]. This has motivated numerous studies on the development of stable neural control strategies for different types of ANN models [38,[40][41][42]44,[50][51][52], including fractional-order neural networks of a continuous and discrete nature [23,24,27,28,30,43,46,47]. For discrete fractional-order neural network models, considerable investigations have been devoted to investigating the Mittag-Lefller stability properties [28,30], which generalize the exponential stability ones.…”