Optimizing Image Acquisition Systems for Autonomous Driving

Blasinski, Henryk; Farrell, Joyce E.; Lian, Trisha; Liu, Zhenyi; Wandell, Brian A.

doi:10.2352/issn.2470-1173.2018.05.pmii-161

Cited by 28 publications

(32 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach synthesizes images as a form of domain adaptation. A third purpose is to co-design cameras and networks [3], [25], [26]. The goal of this application is to explore camera design that spans a wide range parameters, exceeding the consumer photography cameras, in order to discover systems whose images improve the accuracy of object detection networks.…”

Section: Synthetic Data Motivationsmentioning

confidence: 99%

See 1 more Smart Citation

Neural Network Generalization: The Impact of Camera Parameters

et al. 2020

Self Cite

View full text Add to dashboard Cite

We quantify the generalization of a convolutional neural network (CNN) trained to identify cars. First, we perform a series of experiments to train the network using one image dataset -either synthetic or from a camera -and then test on a different image dataset. We show that generalization between images obtained with different cameras is roughly the same as generalization between images from a camera and ray-traced multispectral synthetic images. Second, we use ISETAuto, a soft prototyping tool that creates ray-traced multispectral simulations of camera images, to simulate sensor images with a range of pixel sizes, color filters, acquisition and post-acquisition processing. These experiments reveal how variations in specific camera parameters and image processing operations impact CNN generalization. We find that (a) pixel size impacts generalization, (b) demosaicking substantially impacts performance and generalization for shallow (8-bit) bit-depths but not deeper ones (10-bit), and (c) the network performs well using raw (not demosaicked) sensor data for 10-bit pixels.INDEX TERMS Imaging systems, camera design, autonomous driving, network generalization, convolutional neural network, physically based ray tracing.

show abstract

Section: Synthetic Data Motivationsmentioning

confidence: 99%

“…The CityScape dataset 3 [7] contains 5000 (3700 available) pixel level labeled images collected in street scenes from 50 different cities under daytime. CityScape does not provide camera information for their dataset.…”

Section: ) Cityscapementioning

confidence: 99%

Neural Network Generalization: The Impact of Camera Parameters

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…We advance that work in several ways. The paper by [1] did not include procedural modeling and was restricted to relatively simple scenes; rendering was based on a version of PBRT [26] that was subsequently improved with regards to material modeling. The paper by [24] used Faster RCNN that was pre-trained using camera data from BDD100k [36] and tested on ISET3D synthetic data.…”

Section: Related Workmentioning

confidence: 99%

“…Because these algorithms take time and can be energy consuming, understanding their impact is likely to be an ongoing topic of investigation. To date there are no firm conclusions about their impact [2,1].…”

Section: Image Processingmentioning

confidence: 99%

Soft Prototyping Camera Designs for Car Detection Based on a Convolutional Neural Network

Liu

Lian

Farrell

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

Self Cite

View full text Add to dashboard Cite

Figure 1. Simulated scenes and metadata. An ISE3d generated scene with sky maps measured in (A) early afternoon and (B) late afternoon. The simulation creates pixel-level labeled images including (C) a panoptic segmentation, and (D) a depth map. (E) A simulated scene with superimposed graphs showing the spectral radiance at different image locations (circles). The spectral irradiance is used by the camera simulation to account for the wavelength effects of the lens and color filters. AbstractImaging systems are increasingly used as input to convolutional neural networks (CNN) for object detection; we would like to design cameras that are optimized for this purpose. It is impractical to build different cameras and then acquire and label the necessary data for every potential camera design; creating software simulations of the camera in context (soft prototyping) is the only realistic approach. We implemented soft-prototyping tools that can quantitatively simulate image radiance and camera designs to create realistic images that are input to a convolutional neural network for car detection. We used these methods to quantify the effect that critical hardware components (pixel size), sensor control (exposure algorithms) and image processing (gamma and demosaicing algorithms) have upon average precision of car detection. We quantify (a) the relationship between pixel size and the ability to detect cars at different distances, (b) the penalty for choosing a poor exposure duration, and (c) the ability of the CNN to perform car detection for a variety of post-acquisition processing algorithms. These results show that the optimal choices for car detection are not constrained by the same metrics used for image quality in consumer photography. It is better to evaluate camera designs for CNN applications using soft prototyping with task-specific metrics rather than consumer photography metrics.

show abstract

“…For guiding camera design, it is essential to incorporate real camera parameters when creating synthetic images. Because of this requirement, the simulation must be able to create physically accurate synthetic scene radiance distributions that enable us to model the impact of wavelengthdependent components, including the optics and sensors (Blasinski et al 2018).…”

Section: Introductionmentioning

confidence: 99%

A system for generating complex physically accurate sensor images for automotive applications

Liu¹,

Shen²,

Zhang³

et al. 2019

Self Cite

View full text Add to dashboard Cite

We describe an open-source simulator that creates sensor irradiance and sensor images of typical automotive scenes in urban settings. The purpose of the system is to support camera design and testing for automotive applications. The user can specify scene parameters (e.g., scene type, road type, traffic density, time of day) to assemble a large number of random scenes from graphics assets stored in a database. The sensor irradiance is generated using quantitative computer graphics methods, and the sensor images are created using image systems sensor simulation. The synthetic sensor images have pixel level annotations; hence, they can be used to train and evaluate neural networks for imaging tasks, such as object detection and classification. The end-to-end simulation system supports quantitative assessment -from scene to camera to network accuracy -for automotive applications.

show abstract

Optimizing Image Acquisition Systems for Autonomous Driving

Cited by 28 publications

References 11 publications

Neural Network Generalization: The Impact of Camera Parameters

Neural Network Generalization: The Impact of Camera Parameters

Soft Prototyping Camera Designs for Car Detection Based on a Convolutional Neural Network

A system for generating complex physically accurate sensor images for automotive applications

Contact Info

Product

Resources

About