A Comprehensive Safety Metric to Evaluate Perception in Autonomous Systems

Volk, Georg; Gamerdinger, Jörg; Betnuth, Alexander von; Bringmann, Oliver

doi:10.1109/itsc45102.2020.9294708

Cited by 30 publications

(33 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Volk et al [19] define a combined safety metric for evaluation perception systems that includes a safety component based on the responsibility sensitive safety (RSS, [17]) approach. In contrast to our approach, their detection quality assessment relies on object tracking metrics evaluated over consecutive frames, while we focus on single images.…”

Section: Related Workmentioning

confidence: 99%

“…These encode knowledge about physics and possibilities for future behavior for assessing the situation at hand. As previous works have suggested [11,20,19], we believe that consideration of domain knowledge and the use of task and domain-specific metrics are necessary for evaluating the performance of deep learning-based computer vision functions for safety-critical tasks. As for all production systems, metrics should reflect that the system performs acceptably in all relevant data slices [3].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

From Evaluation to Verification: Towards Task-oriented Relevance Metrics for Pedestrian Detection in Safety-critical Domains

Lyssenko

Gladisch²,

Heinzemann³

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

Whenever a visual perception system is employed in safety-critical applications such as automated driving, a thorough, task-oriented experimental evaluation is necessary to guarantee safe system behavior. While most standard evaluation methods in computer vision provide a good comparability on benchmarks, they tend to fall short on assessing the system performance that is actually relevant for the given task. In our work, we consider pedestrian detection as a highly relevant perception task, and we argue that standard measures such as Intersection over Union (IoU) give insufficient results, mainly because they are insensitive to important physical cues including distance, speed, and direction of motion. Therefore, we investigate so-called relevance metrics, where specific domain knowledge is exploited to obtain a task-oriented performance measure focusing on distance in this initial work. Our experimental setup is based on the CARLA simulator and allows a controlled evaluation of the impact of that domain knowledge. Our first results indicate a linear decrease of the IoU related to the pedestrians' distance, leading to the proposal of a first relevance metric that is also conditioned on the distance.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

From Evaluation to Verification: Towards Task-oriented Relevance Metrics for Pedestrian Detection in Safety-critical Domains

Lyssenko

Gladisch²,

Heinzemann³

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

show abstract

“…Jaccard distance (Volk et al, 2020;Luiten et al, 2021) used both FP and FN instances and is described as:…”

Section: Traditional Metrics or Performance Indicatorsmentioning

confidence: 99%

“…Hence, metrics have been developed to quantify the detection as well as tracking quality (Stiefelhagen et al, 2006). CLassification of Events, Activities and Relationships (CLEAR) is one of the popular studies that described the metrics for quantifying object detection and tracking accuracy (Stiefelhagen et al, 2006;Volk et al, 2020). These metrics can be used for the detection and tracking of obstacles such as pedestrians and vehicles.…”

Section: Clear Metrics For Evaluation Of Object Detection and Trackingmentioning

confidence: 99%

See 1 more Smart Citation

A Literature Review of Performance Metrics of Automated Driving Systems for On-Road Vehicles

Sharath

Mehran

2021

Front. Future Transp.

View full text Add to dashboard Cite

The article presents a review of recent literature on the performance metrics of Automated Driving Systems (ADS). More specifically, performance indicators of environment perception and motion planning modules are reviewed as they are the most complicated ADS modules. The need for the incorporation of the level of threat an obstacle poses in the performance metrics is described. A methodology to quantify the level of threat of an obstacle is presented in this regard. The approach involves simultaneously considering multiple stimulus parameters (that elicit responses from drivers), thereby not ignoring multivariate interactions. Human-likeness of ADS is a desirable characteristic as ADS share road infrastructure with humans. The described method can be used to develop human-like perception and motion planning modules of ADS. In this regard, performance metrics capable of quantifying human-likeness of ADS are also presented. A comparison of different performance metrics is then summarized. ADS operators have an obligation to report any incident (crash/disengagement) to safety regulating authorities. However, precrash events/states are not being reported. The need for the collection of the precrash scenario is described. A desirable modification to the data reporting/collecting is suggested as a framework. The framework describes the precrash sequences to be reported along with the possible ways of utilizing such a valuable dataset (by the safety regulating authorities) to comprehensively assess (and consequently improve) the safety of ADS. The framework proposes to collect and maintain a repository of precrash sequences. Such a repository can be used to 1) comprehensively learn and model the precrash scenarios, 2) learn the characteristics of precrash scenarios and eventually anticipate them, 3) assess the appropriateness of the different performance metrics in precrash scenarios, 4) synthesize a diverse dataset of precrash scenarios, 5) identify the ideal configuration of sensors and algorithms to enhance safety, and 6) monitor the performance of perception and motion planning modules.

show abstract

Simulation-Based Elicitation of Accuracy Requirements for the Environmental Perception of Autonomous Vehicles

Philipp

Qian

Hartjen

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A Comprehensive Safety Metric to Evaluate Perception in Autonomous Systems

Cited by 30 publications

References 16 publications

From Evaluation to Verification: Towards Task-oriented Relevance Metrics for Pedestrian Detection in Safety-critical Domains

From Evaluation to Verification: Towards Task-oriented Relevance Metrics for Pedestrian Detection in Safety-critical Domains

A Literature Review of Performance Metrics of Automated Driving Systems for On-Road Vehicles

Simulation-Based Elicitation of Accuracy Requirements for the Environmental Perception of Autonomous Vehicles

Contact Info

Product

Resources

About