Knowledge Distillation Based on Positive-Unlabeled Classification and Attention Mechanism

Abstract

With the rapid development of deep learning, convolutional neural networks(CNNs) have achieved great success. But these high-capability CNNs often with a huge burden of computation and memory, which hinders these CNNs from applying to practical application. To solve this problem, in this paper, we proposed a method to train a compact model with high-capacity. The student network with fewer parameters and calculations will learning from the knowledge of the teacher network with more parameters and calculations. To promote the ability of the student network, the more expressive knowledge is extracted from the middle-layer feature of neural networks by attention mechanism, and the knowledge transforms more effective from the teacher network to the student network by the positiveunlabeled(PU) classifier. We validate our method in extensive experiments, showing that it can train the student network to achieve significant performance superior to the teacher network.

Complexity-Aware Layer-Wise Mixed-Precision Schemes With SQNR-Based Fast Analysis

“…For efficient DNN acceleration without model accuracy degradation, there have been many studies regarding lightweight deep learning techniques such as network pruning [15], [16], clustering [17], [18], knowledge distillation [19], [20] and hardware optimization [21], [22]. These techniques can be considered for lightweight DNN processing, but it is not easy to apply in practical applications due to their irregularities and dependencies on the neural network [23]. On the other hand, a quantization technique has been proposed as the simplest and most powerful method of lightweight DNN.…”