Exploiting fusion architectures for multispectral pedestrian detection and segmentation

Abstract

Recent research has demonstrated that the fusion of complementary information captured by multi-modal sensors (visible and infrared cameras) enables robust pedestrian detection under various surveillance situations (e.g., daytime and nighttime). In this paper, we investigate a number of fusion architectures in an attempt to identify the optimal way of incorporating multispectral information for joint semantic segmentation and pedestrian detection. We made two important findings: (1) the sum fusion strategy, which computes the sum of two feature maps at the same spatial locations, delivers the best performance of multispectral detection, while the most commonly used concatenation fusion surprisingly performs the worst; and (2) two-stream semantic segmentation without multispectral fusion is the most effective scheme to infuse semantic information as supervision for learning human-related features. Based on these studies, we present a unified multispectral fusion framework for joint training of semantic segmentation and target detection that outperforms state-of-the-art multispectral pedestrian detectors by a large margin on the KAIST benchmark dataset.

Box-level segmentation supervised deep neural networks for accurate and real-time multispectral pedestrian detection

“…3 Our method achieves significantly higher detection accuracy compared with the state-of-the-art multispectral pedestrian detectors [27,24,16,15,31]. Moreover, this efficient framework can process more than 30 images per second on a single NVIDIA Geforce Titan X GPU to facilitate real-time applications in autonomous vehicles.…”

“…However, the use of anchor boxes will cause severe imbalance between positive and negative training samples [37] and involve complex hyperparameter settings (e.g., box size, aspect ratio, stride, and intersection-over-union threshold) [29]. Our method is very different from the existing anchor box based multispectral pedestrian detectors [27,24,32,16,15,31] in two major aspects. Firstly, we make use of the ground truth bounding boxes (manually annotated) to generate coarse boxlevel segmentation masks, which are utilized to replace the anchor bounding boxes for the training of two-stream deep neural networks to learn human-relative characteristic features.…”

“…Recently, researchers explore illumination information of a scene and proposed illumination-aware weighting mechanism to boost multispectral pedestrian detection performances [16,32]. Guan et al [15] presented a unified multispectral fusion framework for joint training of semantic segmentation and target detection. More accurate detection results were obtained by infusing the multispectral semantic segmentation masks as supervision for learning human-related features.…”

Unsupervised Domain Adaptation for Multispectral Pedestrian Detection

“…Researchers also paid attention to the main difference between visible and infrared images, and proposed illumination-aware weighting mechanism to give extra information to detectors [10,17]. Guan et al [9] presented a unified multispectral fusion framework, which infuses the multispectral semantic segmentation masks as supervision for learning human-related features, getting more accurate detection results. Li et al [16] designed a cascaded multispectral classification network to distinguish hard negatives sample from pedestrian and human-like instances.…”

“…Inspired by the multi-task framework for joint training of multispectral pedestrian detection and semantic segmentation [9], we combine the visible and thermal pedestrian detection supervision module with the box-level segmentation supervised deep neural networks [3] to build multispectral pedestrian detector, as illustrated in Fig. 3.…”