Generalization ability of region proposal networks for multispectral person detection

Abstract

Multispectral person detection aims at automatically localizing humans in images that consist of multiple spectral bands. Usually, the visual-optical (VIS) and the thermal infrared (IR) spectra are combined to achieve higher robustness for person detection especially in insufficiently illuminated scenes. This paper focuses on analyzing existing detection approaches for their generalization ability. Generalization is a key feature for machine learning based detection algorithms that are supposed to perform well across different datasets. Inspired by recent literature regarding person detection in the VIS spectrum, we perform a cross-validation study to empirically determine the most promising dataset to train a well-generalizing detector. Therefore, we pick one reference Deep Convolutional Neural Network (DCNN) architecture as well as three different multispectral datasets. The Region Proposal Network (RPN) that was originally introduced for object detection within the popular Faster R-CNN is chosen as a reference DCNN. The reason for this choice is that a stand-alone RPN is able to serve as a competitive detector for two-class problems such as person detection. Furthermore, all current state-of-theart approaches initially apply an RPN followed by individual classifiers. The three considered datasets are the KAIST Multispectral Pedestrian Benchmark including recently published improved annotations for training and testing, the Tokyo Multi-spectral Semantic Segmentation dataset, and the OSU Color-Thermal dataset including just recently released annotations. The experimental results show that the KAIST Multispectral Pedestrian Benchmark with its improved annotations provides the best basis to train a DCNN with good generalization ability compared to the other two multispectral datasets. On average, this detection model achieves a log-average Miss Rate (MR) of 29.74 % evaluated on the reasonable test subsets of the three analyzed datasets.

IR-MSDNet: Infrared and Visible Image Fusion Based On Infrared Features and Multiscale Dense Network

“…Furthermore, to emphasis the effectiveness of IR-MSDNet, it is further applied to object detection of remote sensing. RPN [2] is chosen as the basic state-of-the-art detector on KAIST [42] Multispectral (IR, Visible) benchmark datasets. Log average Miss Rate (MR) metric [2] as the standard measure for object detection is used for quantitative evaluation.…”

“…RPN [2] is chosen as the basic state-of-the-art detector on KAIST [42] Multispectral (IR, Visible) benchmark datasets. Log average Miss Rate (MR) metric [2] as the standard measure for object detection is used for quantitative evaluation. Log average MR is computed by averaging the miss rates over different false positive per-image (FPPI) points sampled within the evenly spaced in log-space.…”

“…In Table IV, RPN detector is used to detect different objects from visible images, infrared images, fused images generated by different fusion methods, i.e. JSRSD [4], VggML [9], DeepFuse [8], FusionGAN [11], DenseFuse [12], IFCNN [10], two streams (feature fusion from visible and infrared image) [2], and our proposed IR-MSDNet, respectively. It can be seen from Table IV, the lowest average value of 27.45% is achieved by the proposed IR-MSDNet, which means that the proposed IR-MSDNet has greatly enhanced the details in the fused images and makes the RPN detector obtain the best object detection results.…”

“…EMOTE sensing image fusion has been studied for decades, because complementary information from multisource remote sensing images in the fused image is of great help to various remote sensing applications such as surveillance, object detection (OD) and etc. so on [1], [2]. The fusion of infrared image and visible image is one of the important tasks in remote sensing field.…”

The Effect of a Diverse Dataset for Transfer Learning in Thermal Person Detection

“…They tested their algorithm on the OSU color thermal dataset [20],video analytic dataset [21], and LITIVdataset [22]. Fritz et al [23] investigated the generalization of a deep learning network in multispectral person detection datasets. They mainly used the Caltech [24], city person [25], CVC-09 [26], KAIST [1], OSU color thermal [20], and Tokyo segmentation [27] datasets for their investigation.…”