Testing Deep Learning-based Visual Perception for Automated Driving

Abrecht, Stephanie; Gauerhof, Lydia; Gladisch, Christoph; Groh, Konrad; Heinzemann, Christian; Woehrle, Matthias

doi:10.1145/3450356

Cited by 14 publications

(8 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This set of claims evaluates individual properties P that are required in order to minimize the safety-related performance insufficiencies in the model. The estimated failure rate with respect to different properties P may be estimated using testing techniques or with formal verification (Huang et al, 2020;Abrecht et al, 2021). Formal verification can include an exhaustive exploration of a bounded hypersphere defining the vicinity of particular samples to demonstrate local robustness properties (Cheng et al, 2017;Huang et al, 2017) and several techniques have been put forward to apply constraint solving to this problem.…”

Section: Evaluation Of Performancementioning

confidence: 99%

Addressing uncertainty in the safety assurance of machine-learning

Burton

Herd

2023

Front. Comput. Sci.

View full text Add to dashboard Cite

There is increasing interest in the application of machine learning (ML) technologies to safety-critical cyber-physical systems, with the promise of increased levels of autonomy due to their potential for solving complex perception and planning tasks. However, demonstrating the safety of ML is seen as one of the most challenging hurdles to their widespread deployment for such applications. In this paper we explore the factors which make the safety assurance of ML such a challenging task. In particular we address the impact of uncertainty on the confidence in ML safety assurance arguments. We show how this uncertainty is related to complexity in the ML models as well as the inherent complexity of the tasks that they are designed to implement. Based on definitions of uncertainty as well as an exemplary assurance argument structure, we examine typical weaknesses in the argument and how these can be addressed. The analysis combines an understanding of causes of insufficiencies in ML models with a systematic analysis of the types of asserted context, asserted evidence and asserted inference within the assurance argument. This leads to a systematic identification of requirements on the assurance argument structure as well as supporting evidence. We conclude that a combination of qualitative arguments combined with quantitative evidence are required to build a robust argument for safety-related properties of ML functions that is continuously refined to reduce residual and emerging uncertainties in the arguments after the function has been deployed into the target environment.

show abstract

Section: Evaluation Of Performancementioning

confidence: 99%

Addressing uncertainty in the safety assurance of machine-learning

Burton

Herd

2023

Front. Comput. Sci.

View full text Add to dashboard Cite

show abstract

“…For the white box coverage criteria, neuron coverage [12] and extensions (e.g., SS-coverage [15]) motivated by MC/DC coverage in classical software have been proposed. For the black box coverage criteria, multiple results are utilizing combinatorial testing [3,1] to argue about the relative completeness of the test data. Readers may reference Section 5.1 of a recent survey paper [6] for an overview of existing results in coverage-driven testing.…”

Section: Related Workmentioning

confidence: 99%

Unaligned but Safe -- Formally Compensating Performance Limitations for Imprecise 2D Object Detection

Schuster¹,

Seferis²,

Burton³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we consider the imperfection within machine learning-based 2D object detection and its impact on safety. We address a special sub-type of performance limitations: the prediction bounding box cannot be perfectly aligned with the ground truth, but the computed Intersection-over-Union metric is always larger than a given threshold. Under such type of performance limitation, we formally prove the minimum required bounding box enlargement factor to cover the ground truth. We then demonstrate that the factor can be mathematically adjusted to a smaller value, provided that the motion planner takes a fixed-length buffer in making its decisions. Finally, observing the difference between an empirically measured enlargement factor and our formally derived worst-case enlargement factor offers an interesting connection between the quantitative evidence (demonstrated by statistics) and the qualitative evidence (demonstrated by worst-case analysis).

show abstract

“…Most notably, if researchers have no understanding of software execution, it could be the only practicable method [6]. Undoubtedly, due to the change in advancement framework caused by ML, this feature is also an essential advantage for evaluating ML systems [7]. Conventional software is built axiomatically by attempting to write down the regulations as program codes, and the system's behavior is controlled by these known regulations.…”

Section: Fig 1 Representation Of Testing Using Machine Learningmentioning

confidence: 99%

Software Defect Prediction using Deep Learning by Correlation Clustering of Testing Metrics

Sharma

Sinha

Sharma

2022

Int. j. electr. comput. eng. syst. (Online)

View full text Add to dashboard Cite

The software industry has made significant efforts in recent years to enhance software quality in businesses. The use of proactively defect prediction in the software will assist programmers and white box testing in detecting issues early, saving time and money. Conventional software defect prediction methods focus on traditional source code metrics such as code complexities, lines of code, and so on. These capabilities, unfortunately, are unable to retrieve the semantics of source code. In this paper, we have presented a novel Correlation Clustering fine-tuned CNN (CCFT-CNN) model based on testing Metrics. CCFT-CNN can predict the regions of source code that contain faults, errors, and bugs. Abstract Syntax Tree (AST) tokens are extracted as testing Metrics vectors from the source code. The correlation among AST testing Metrics is performed and clustered as a more relevant feature vector and fed into Convolutional Neural Network (CNN). Then, to enhance the accuracy of defect prediction, fine-tuning of the CNN model is performed by applying hyperparameters. The result analysis is performed on the PROMISE dataset that contains samples of open-source Java applications such as Camel Dataset, Jedit dataset, Poi dataset, Synapse dataset, Xerces dataset, and Xalan dataset. The result findings show that the CCFT- CNN model increases the average F-measure by 2% when compared to the baseline model.

show abstract

Testing Deep Learning-based Visual Perception for Automated Driving

Cited by 14 publications

References 59 publications

Addressing uncertainty in the safety assurance of machine-learning

Addressing uncertainty in the safety assurance of machine-learning

Unaligned but Safe -- Formally Compensating Performance Limitations for Imprecise 2D Object Detection

Software Defect Prediction using Deep Learning by Correlation Clustering of Testing Metrics

Contact Info

Product

Resources

About