Deep Visible and Thermal Camera-based Optimal Semantic Segmentation using Semantic Forecasting

Abstract

Visible camera-based semantic segmentation and semantic forecasting are important perception tasks in autonomous driving. In semantic segmentation, the current frame's pixel level labels are estimated using the current visible frame. In semantic forecasting, the future frame's pixel-level labels are predicted using the current and the past visible frames and pixel-level labels. While reporting state-of-the-art accuracy, both of these tasks are limited by the visible camera's susceptibility to varying illumination, adverse weather conditions, sunlight and headlight glare etc. In this work, we propose to address these limitations using the deep sensor fusion of the visible and the thermal camera. The proposed sensor fusion framework performs both semantic forecasting as well as an optimal semantic segmentation within a multi-step iterative framework. In the first or forecasting step, the framework predicts the semantic map for the next frame. The predicted semantic map is updated in the second step, when the next visible and thermal frame is observed. The updated semantic map is considered as the optimal semantic map for the given visible-thermal frame. The semantic map forecasting and updating are iteratively performed over time. The estimated semantic maps contain the pedestrian behavior, the free space and the pedestrian crossing labels. The pedestrian behavior is categorized based on their spatial, motion and dynamic orientation information. The proposed framework is validated using the public KAIST dataset. A detailed comparative analysis and ablation study is performed using pixel-level classification and IOU error metrics. The results show that the proposed framework can not only accurately forecast the semantic segmentation map but also accurately update them.

Perception and Sensing for Autonomous Vehicles Under Adverse Weather Conditions: A Survey

“…Their fused image is produced with the colors from the RGB image and the details from the NIR. John et al [127] also proposed a visible and thermal camera deep sensor fusion framework that performs both semantic accurate forecasting as well as optimal semantic segmentation. These might be some of the most cost-effective solutions for weather conditions but particular gated CMOS imaging systems are still being developed [59].…”

Spatio-Temporal Information for Action Recognition in Thermal Video Using Deep Learning Model

“…Substantial improvement has been achieved in action detection over the last several decades, and the majority of existing techniques for action classification have been used in visible image clips. [1][2][3]. In addition, several visible light action datasets, such as UCF101, KTH and HMDB51 have been developed for action detection.…”