RGB Cameras Failures and Their Effects in Autonomous Driving Applications

Secci, Francesco; Ceccarelli, Andrea

doi:10.1109/tdsc.2022.3156941

Cited by 15 publications

(9 citation statements)

References 60 publications

(86 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most comprehensive work on visual camera failures is [22], where the authors systematically identified the failure modes and effects of a visual camera through the application of a Failure Mode and Effects Analysis (FMEA). The effects of camera failures on the output images are summarized in Fig.…”

Section: Background On Visual Camera Failuresmentioning

confidence: 99%

“…multitude of reasons [32]: this paper accounts for possible failures of the visual camera, which may be due to internal (e.g., failures of the electrical parts), external (e.g., dirt or scratched lenses), or environmental (e.g., rain or icing on lenses) factors. We consider a complete set of visual camera failures resulting from a Failure Mode and Effect Analysis [22] to embrace a complete set of visual camera failures known to date, according to the literature.…”

Section: Introductionmentioning

confidence: 99%

“…We assessed this methodology by considering the deployment of image classifiers in autonomous vehicles for Traffic Sign Recognition (TSR), which may be significantly affected by visual camera failures [22], [28], [33]. We gathered three well-known TSR datasets, namely the German Traffic Sign Recognition Benchmark (GTSRB, [10]), the Belgium Traffic Sign (BelgiumTSC, [11]), and the Dataset of Italian Traffic Signs (DITS, [12]), applied AlexNet [7], MobileNetV2 [9], and Inceptionv3 [8] DNNs, which have wide application in the TSR domain [3], [4], [5], and injected a total of 13 visual camera failures under different configurations.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Tolerate Failures of the Visual Camera With Robust Image Classifiers

et al. 2023

Self Cite

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) have become an enabling technology for building accurate image classifiers, and are increasingly being applied in many ICT systems such as autonomous vehicles. Unfortunately, classifiers can be deceived by images that are altered due to failures of the visual camera, preventing the proper execution of the classification process. Therefore, it is of utmost importance to build image classifiers that can guarantee accurate classification even in the presence of such camera failures. This study crafts classifiers that are robust to failures of the visual camera by augmenting the training set with artificially altered images that simulate the effects of such failures. Such a data augmentation approach improves classification accuracy with respect to the most common data augmentation approaches, even in the absence of camera failures. To provide experimental evidence for our claims, we exercise three DNN image classifiers on three image datasets, in which we inject the effects of many failures into the visual camera. Finally, we applied eXplainable AI to debate why classifiers trained with the data augmentation approach proposed in this study can tolerate failures of the visual camera.

show abstract

Section: Background On Visual Camera Failuresmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Tolerate Failures of the Visual Camera With Robust Image Classifiers

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…Therefore, we consider faults in the three cameras used in the autonomous vehicle in our case study (discussed in the experiments section). While many types of faults can be associated with a digital camera, we focus on the common fault of occlusion [5]. We train a monitor using prior data and only use inference on the trained models at decision time.…”

Section: Runtime Monitorsmentioning

confidence: 99%

Dynamic Simplex: Balancing Safety and Performance in Autonomous Cyber Physical Systems

Luo¹,

Ramakrishna²,

Pettet³

et al. 2023

Preprint

View full text Add to dashboard Cite

Learning Enabled Components (LEC) have greatly assisted cyberphysical systems in achieving higher levels of autonomy. However, LEC's susceptibility to dynamic and uncertain operating conditions is a critical challenge for the safety of these systems. Redundant controller architectures have been widely adopted for safety assurance in such contexts. These architectures augment LEC "performant" controllers that are difficult to verify with "safety" controllers and the decision logic to switch between them. While these architectures ensure safety, we point out two limitations. First, they are trained offline to learn a conservative policy of always selecting a controller that maintains the system's safety, which limits the system's adaptability to dynamic and non-stationary environments. Second, they do not support reverse switching from the safety controller to the performant controller, even when the threat to safety is no longer present. To address these limitations, we propose a dynamic simplex strategy with an online controller switching logic that allows two-way switching. We consider switching as a sequential decision-making problem and model it as a semi-Markov decision process. We leverage a combination of a myopic selector using surrogate models (for the forward switch) and a non-myopic planner (for the reverse switch) to balance safety and performance. We evaluate this approach using an autonomous vehicle case study in the CARLA simulator using different driving conditions, locations, and component failures. We show that the proposed approach results in fewer collisions and higher performance than state-ofthe-art alternatives.

show abstract

“…In recent years, depth cameras have been widely utilized in various fields, such as autonomous driving [1][2][3], robot grasping [4][5][6], and simultaneous localization and mapping (SLAM) [7][8][9]. However, due to object material, specular reflection, or occlusion, the captured depth data often exhibits holes at the edges where objects come into contact with the background.…”

Section: Introductionmentioning

confidence: 99%

Non-Local Means Hole Repair Algorithm Based on Adaptive Block

Zhao,

Li,

Pan

2023

Applied Sciences

View full text Add to dashboard Cite

RGB-D cameras provide depth and color information and are widely used in 3D reconstruction and computer vision. In the majority of existing RGB-D cameras, a considerable portion of depth values is often lost due to severe occlusion or limited camera coverage, thereby adversely impacting the precise localization and three-dimensional reconstruction of objects. In this paper, to address the issue of poor-quality in-depth images captured by RGB-D cameras, a depth image hole repair algorithm based on non-local means is proposed first, leveraging the structural similarities between grayscale and depth images. Second, while considering the cumbersome parameter tuning associated with the non-local means hole repair method for determining the size of structural blocks for depth image hole repair, an intelligent block factor is introduced, which automatically determines the optimal search and repair block sizes for various hole sizes, resulting in the development of an adaptive block-based non-local means algorithm for repairing depth image holes. Furthermore, the proposed algorithm’s performance are evaluated using both the Middlebury stereo matching dataset and a self-constructed RGB-D dataset, with performance assessment being carried out by comparing the algorithm against other methods using five metrics: RMSE, SSIM, PSNR, DE, and ALME. Finally, experimental results unequivocally demonstrate the innovative resolution of the parameter tuning complexity inherent in-depth image hole repair, effectively filling the holes, suppressing noise within depth images, enhancing image quality, and achieving elevated precision and accuracy, as affirmed by the attained results.

show abstract

RGB Cameras Failures and Their Effects in Autonomous Driving Applications

Cited by 15 publications

References 60 publications

Tolerate Failures of the Visual Camera With Robust Image Classifiers

Tolerate Failures of the Visual Camera With Robust Image Classifiers

Dynamic Simplex: Balancing Safety and Performance in Autonomous Cyber Physical Systems

Non-Local Means Hole Repair Algorithm Based on Adaptive Block

Contact Info

Product

Resources

About