ISETAuto: Detecting Vehicles With Depth and Radiance Information

Abstract

Supported by Jilin University. We thank Boyd Fowler at Omnivision and Sergio Goma at Qualcomm for drawing our attention to prior work on RGB-D sensor technology.

Accurate smartphone camera simulation using 3D scenes

“…For example, neural networks trained on physically-based simulations of camera images can detect cars in real camera image data almost as well as neural networks trained on the real camera image data [38]. Furthermore, neural networks trained on camera images generated by physically-based simulations performed better than neural networks trained on camera images generated by raster-based graphics or by ray-traced graphics rendering methods that did not include the correct optics and sensor modeling [2], [12], [38], [49].…”

“…In one study, we explored the effect that sensor pixel size has on performance [38]. And, in another study, we compared the performance of imaging sensors that are capable of collecting radiance or depth information, and a hybrid imaging sensor that can capture both radiance and depth simultaneously on the same sensor array [2].…”

“…Section and VII describes ongoing and future work. The image systems simulation validation includes three main parts: (1) a 3D scene radiance description that models light sources, asset geometry, and material properties, (2) an optical model that maps rays from the scene onto the sensor, and (3) a model of how the irradiance at the sensor is converted into unprocessed digital values.…”

“…Image system simulations model how properties of the scene, optical components, and sensor interact to produce the final sensor image. These end-toend simulations can be used to design rendering methods for consumer photography or to develop machine learning algorithms [1], [2].…”

Validation of image systems simulation technology using a Cornell Box

“…We are using end-to-end image systems simulations methods for a wide range of applications. These include designing and evaluating novel sensor designs for high dynamic range imaging [19], underwater imaging [20], autonomous driving [21] and fluorescence imaging [10,11]. End-to-end image systems simulations can accelerate innovation by reducing many of the timeconsuming and expensive steps in designing, building and evaluating image systems.…”