High-Dynamic-Range Tone Mapping in Intelligent Automotive Systems

Abstract

Intelligent driver assistance systems are becoming increasingly popular in modern passenger vehicles. A crucial component of intelligent vehicles is the ability to detect vulnerable road users (VRUs) for an early and safe response. However, standard imaging sensors perform poorly in conditions of strong illumination contrast, such as approaching a tunnel or at night, due to their dynamic range limitations. In this paper, we focus on the use of high-dynamic-range (HDR) imaging sensors in vehicle perception systems and the subsequent need for tone mapping of the acquired data into a standard 8-bit representation. To our knowledge, no previous studies have evaluated the impact of tone mapping on object detection performance. We investigate the potential for optimizing HDR tone mapping to achieve a natural image appearance while facilitating object detection of state-of-the-art detectors designed for standard dynamic range (SDR) images. Our proposed approach relies on a lightweight convolutional neural network (CNN) that tone maps HDR video frames into a standard 8-bit representation. We introduce a novel training approach called detection-informed tone mapping (DI-TM) and evaluate its performance with respect to its effectiveness and robustness in various scene conditions, as well as its performance relative to an existing state-of-the-art tone mapping method. The results show that the proposed DI-TM method achieves the best results in terms of detection performance metrics in challenging dynamic range conditions, while both methods perform well in typical, non-challenging conditions. In challenging conditions, our method improves the detection F2 score by 13%. Compared to SDR images, the increase in F2 score is 49%.

Deep Learning Tone-Mapping and Demosaicing for Automotive Vision Systems

“…Although there is at least one study that performs object detection on HDR content, the main SOTA object detection algorithms on which improvements are constantly being made are trained on 8-bit content. A training approach that uses a lightweight CNN architecture for tone-mapping with the intention, for the first time, of increasing the detectability of VRUs in reconstructed 8-bit content, is proposed in [2]. The results of a comparison with one of the classical state-of-the-art algorithms for tone-mapping show that this approach, by targeting the important aspects of tone-mapping in various challenging conditions and by performing a region-of-interest (ROI) selective training, succeeds in increasing the detectability of the VRUs in both challenging and non-challenging cases.…”

“…SOTA algorithms for tone-mapping, object detection and demosaicing mainly exist and have progressed independently from each other. With the performed research, which builds upon the research presented in [2], we address the common ADS problem they are trying to solve, which is improving object detection in challenging traffic scenes. With the proposed research, we attempt to increase the detectability of the traffic-related objects in traffic scenes with varying degrees of illumination, contrast, weather and daylight ambient conditions by performing optimal tone-mapping and demosaicing on HDR content.…”

“…The first network, "CNN for TM", is devised solely for tonemapping where a full-color, full-range HDR input is used. This network builds upon the architecture and the methodology presented in [2]. This network is used in the sequential demosaicing and tone-mapping pipeline.…”

“…Then, we present the main novelties of the performed research. Additionally, we briefly describe the architecture of the neural network (devised for inversetone-mapping) that was used as starting point architecture for the approach presented in [2], upon which we build our research. In Section 3, we propose two CNN architectures and describe their design and the applied training procedures in more detail.…”

A Low-Latency Noise-Aware Tone Mapping Operator for Hardware Implementation with a Locally Weighted Guided Filter

“…For instance, Refs. [5][6][7] introduced TMOs based on neural networks, while [8][9][10] presented the hardware implementations optimized for different TMOs. Therefore, research on TMOs and their hardware implementation remains an active and important research field.…”