Distilling Object Detectors with Task Adaptive Regularization

Abstract

Current state-of-the-art object detectors are at the expense of high computational costs and are hard to deploy to low-end devices. Knowledge distillation, which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the promising solutions for model miniaturization. In this paper, we investigate each module of a typical detector in depth, and propose a general distillation framework that adaptively transfers knowledge from teacher to student according to the task specific priors. The intuition is that simply distilling all information from teacher to student is not advisable, instead we should only borrow priors from the teacher model where the student cannot perform well. Towards this goal, we propose a region proposal sharing mechanism to interflow region responses between the teacher and student models. Based on this, we adaptively transfer knowledge at three levels, i.e., feature backbone, classification head, and bounding box regression head, according to which model performs more reasonably. Furthermore, considering that it would introduce optimization dilemma when minimizing distillation loss and detection loss simultaneously, we propose a distillation decay strategy to help improve model generalization via gradually reducing the distillation penalty. Experiments on widely used detection benchmarks demonstrate the effectiveness of our method. In particular, using Faster R-CNN with FPN as an instantiation, we achieve an accuracy of 39.0% with Resnet-50 on COCO dataset, which surpasses the baseline 36.3% by 2.7% points, and even better than the teacher model with 38.5% mAP.

Focal and Global Knowledge Distillation for Detectors

“…Wang et al [28] propose the finegrained mask to distill the regions calculated by groundtruth bounding boxes. Sun et al [25] utilize the Gaussian Mask to cover the ground-truth for distillation. Such methods lack the distillation for the background.…”

“…Mimick [15] distills the positive area proposed by region proposal network (RPN) of the student detector. FGFI [28] and TADF [25] use the fine-grained and Gaussian Mask to select the distillation area, respectively. Defeat [7] distills the foreground and background separately.…”

Prediction-Guided Distillation for Dense Object Detection

“…KD was first popularised for image classification [14] where a student model is trained to mimic the soft labels generated by a teacher model. However, this [10,30,34], Our approach (d) focuses on a few key predictive regions of the teacher.…”

“…While soft label-based KD can be directly applied for classification, finding an equivalent for localisation remains a challenge. Recent work [9,10,30,34,35,37,41] alleviates this problem by forcing the student model to generate feature maps similar to the teacher counterpart; a process known as feature imitation.…”

“…This question is of the utmost importance for dense object detectors [7,16,19,31,38,42] because, unlike two-stage detectors [3,12,27], they do not use the RoIAlign [12] operation to explicitly pool and align object features; instead they output predictions at every location of the feature map [17]. Recent work [30,35] has shown that distilling the whole feature map with equal weighting is sub-optimal because not all features carry equally meaningful information. Therefore, a weighting mechanism that assigns appropriate importance to different regions, particularly to foreground regions near the objects, is highly desirable for dense object detectors, and has featured in recent work.…”