Multimodal object detection using unsupervised transfer learning and adaptation techniques

Abstract

Deep neural networks achieve state-of-the-art performance on object detection tasks with RGB data. However, there are many advantages of detection using multi-modal imagery for defence and security operations. For example, the IR modality offers persistent surveillance and is essential in poor lighting conditions and 24hr operation. It is, therefore, crucial to create an object detection system which can use IR imagery. Collecting and labelling large volumes of thermal imagery is incredibly expensive and time-consuming. Consequently, we propose to mobilise labelled RGB data to achieve detection in the IR modality. In this paper, we present a method for multi-modal object detection using unsupervised transfer learning and adaptation techniques. We train faster RCNN on RGB imagery and test with a thermal imager. The images contain object classes; people and land vehicles and represent real-life scenes which include clutter and occlusions. We improve the baseline F1-score by up to 20% through training with an additional loss function, which reduces the difference between RGB and IR feature maps. This work shows that unsupervised modality adaptation is possible, and we have the opportunity to maximise the use of labelled RGB imagery for detection in multiple modalities. The novelty of this work includes; the use of the IR imagery, modality adaption from RGB to IR for object detection and the ability to use real-life imagery in uncontrolled environments. The practical impact of this work to the defence and security community is an increase in performance and the saving of time and money in data collection and annotation.

Unsupervised object detection via LWIR/RGB translation

“…However, fine-tuning is not always possible due to lack of labelled data. Paper [2] propose a novel architecture for LWIR detection in an unsupervised manner for the first time. They use adaptation techniques (previously used within the RGB domain for classification tasks) for creating modality invariant features in a faster RCNN network for improving LWIR detection.…”

“…Since we aim to find where the objects are in an unsupervised manner, without requiring any label, we use the output of the detection algorithm rather than ground truth. The real LWIR and RGB images pass through an RGB-trained faster RCNN network adapted for LWIR in an unsupervised manner described in [2]. Although this network is adapted for LWIR imagery, it maintains its RGB performance, so it can be used in both modalities simultaneously.…”

300 GHz radar object recognition based on deep neural networks and transfer learning

“…Abbott, et al [8] uses RGB and infrared cameras to recognise pedestrians ( Figure 2.44). It employs a loss function which uses output from two neural network from both sensors, thus leading to a better detection rate.…”

“…This project aims to develop an autonomous car perception system combining low-THz radar, video and LiDAR for adverse weather scenarios. This thesis was developed as part of the Pervasive low-TeraHz and Video Sensing for Car Autonomy and Driver Assistance project (PATH CAD), which is a collaboration between Heriot-Watt University, Infrared is an alternative sensor for day and night perception [6][7][8]. Objects emit heat radiation based on their temperature.…”

“…Methodology developed by Abbott et al[8].range capabilities of radar sensors. With the increased development of the radar technology, radar was began to be used in automotive scenarios.…”