Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrum

Abstract

ImageNet pre-training has been regarded as essential for training accurate object detectors for a long time. Recently, it has been shown that object detectors trained from randomly initialized weights can be on par with those finetuned from ImageNet pre-trained models. However, the effects of pre-training and the differences caused by pretraining are still not fully understood. In this paper, we analyze the eigenspectrum dynamics of the covariance matrix of each feature map in object detectors. Based on our analysis on ResNet-50, Faster R-CNN with FPN, and Mask R-CNN, we show that object detectors trained from Ima-geNet pre-trained models and those trained from scratch behave differently from each other even if both object detectors have similar accuracy. Furthermore, we propose a method for automatically determining the widths (the numbers of channels) of object detectors based on the eigenspectrum. We train Faster R-CNN with FPN from randomly initialized weights, and show that our method can reduce ∼27% of the parameters of ResNet-50 without increasing Multiply-Accumulate operations and losing accuracy. Our results indicate that we should develop more appropriate methods for transferring knowledge from image classification to object detection (or other tasks).

DAP: Detection-Aware Pre-training with Weak Supervision

“…For example, in the right part of Figure 5, 500 fine-tuning iterations from DAP already achieve ≥ 65 AP .5 , while the corresponding CLS numbers are lower than 20. This demonstrates that a better pre-trained model can provide faster convergence speed, which is consistent with [25,20,16,34].…”

“…Classification pre-training may sometimes even harm localization when the downstream data is abundant while benefit classification [25]. Shinya et al try to understand the impact of ImageNet classification pre-training on detection and discover that the pre-trained model generates narrower eigenspectrum than the fromscratch model [34]. Recent work proposes a cheaper Montage pre-training for detection on the target detection data and obtains an on-par or better performance than ImageNet classification pre-training [48].…”

An Educated Warm Start For Deep Image Prior-Based Micro CT Reconstruction

“…In particular, the number of singular values larger than a given (plausible) threshold changes consistently. A similar observation was also made in [53] in the context of object detectors (but based on the normalized eigenspectrum of the Hessian). This observation may explain the different dynamics of the optimization scheme for the pretrained model and the model trained from scratch.…”

USB: Universal-Scale Object Detection Benchmark

“…Epochs for the first learning rate decay, the second decay, and ending training are (8,11,12) for the 1× schedule, (16,22,24) for the 2× schedule, and (16,19,20) for the 20e schedule. To avoid overfitting by small learning rates [52], the 20e schedule is reasonable.…”