Deep Face Recognition Model Compression via Knowledge Transfer and Distillation

Abstract

Fully convolutional networks (FCNs) have become de facto tool to achieve very high-level performance for many vision and non-vision tasks in general and face recognition in particular. Such high-level accuracies are normally obtained by very deep networks or their ensemble. However, deploying such high performing models to resource constraint devices or real-time applications is challenging. In this paper, we present a novel model compression approach based on student-teacher paradigm for face recognition applications. The proposed approach consists of training teacher FCN at bigger image resolution while student FCNs are trained at lower image resolutions than that of teacher FCN. We explored three different approaches to train student FCNs: knowledge transfer (KT), knowledge distillation (KD) and their combination. Experimental evaluation on LFW and IJB-C datasets demonstrate comparable improvements in accuracies with these approaches. Training low-resolution student FCNs from higher resolution teacher offer fourfold advantage of accelerated training, accelerated inference, reduced memory requirements and improved accuracies. We evaluated all models on IJB-C dataset and achieved state-of-the-art results on this benchmark. The teacher network and some student networks even achieved Top-1 performance on IJB-C dataset. The proposed approach is simple and hardware friendly, thus enables the deployment of high performing face recognition deep models to resource constraint devices.

A Lightweight Identification Method for Complex Power Industry Tasks Based on Knowledge Distillation and Network Pruning

“…In addition, employing a loop training strategy to train multiple networks simultaneously and weakening the relationship between instruction and learning can further optimize the teacher-student strategy [18]. To address the problem of low resolution, a teacher-student strategy using the same architecture was proposed in [19] which can be applied to train images with different resolutions. Also, some improvements have been made to the traditional distillation methods for the semantic segmentation task, such as using an association adaptation module to enable the student model to obtain and extract more information when learning about the teacher's knowledge [20].…”

Teacher Guided Neural Architecture Search for Face Recognition

“…For face recognition knowledge distillation, there have been several attempts (Wang, Lan, and Zhang 2017;Luo et al 2016;Karlekar, Feng, and Pranata 2019;Ge et al 2018;Feng et al 2019;Peng et al 2019;Wang et al 2019aWang et al , 2020a in literatures to distil large CNNs, so as to make their deployments easier. Hinton et al (Hinton, Vinyals, and Dean.…”

“…Luo et al (Luo et al 2016) propose a neuron selection method by leveraging the essential characteristics (domain knowledge) of the learned face representation. Karlekar et al (Karlekar, Feng, and Pranata 2019) simultaneously exploit one-hot labels and feature vectors for the knowledge transfer between different face resolutions. Ge et al (Ge et al 2018) develop a selective knowledge distillation, which selectively distils the most informative facial features by solving a sparse graph optimization problem.…”