Evaluation of Virtual Traffic Situations for Testing Automated Driving Functions based on Multidimensional Criticality Analysis

Huber, Beate; Herzog, Steffen; Sippl, Christoph; German, Reinhard; Djanatliev, Anatoli

doi:10.1109/itsc45102.2020.9294169

Cited by 7 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An initial overview on contemporary criticality metrics has been published by the authors and is extended and unified in this work [69, p. 14]. Similarly, there exists work within the AV community that refers to a certain subset of criticality metrics, but does not provide an extensive review on the possible choices [31,36]. Additionally, motion models play a crucial role when assessing future evaluations of a scene.…”

Section: Related Workmentioning

confidence: 99%

Criticality Metrics for Automated Driving: A Review and Suitability Analysis of the State of the Art

Westhofen

Neurohr

Koopmann

et al. 2022

Arch Computat Methods Eng

View full text Add to dashboard Cite

The large-scale deployment of automated vehicles on public roads has the potential to vastly change the transportation modalities of today’s society. Although this pursuit has been initiated decades ago, there still exist open challenges in reliably ensuring that such vehicles operate safely in open contexts. While functional safety is a well-established concept, the question of measuring the behavioral safety of a vehicle remains subject to research. One way to both objectively and computationally analyze traffic conflicts is the development and utilization of so-called criticality metrics. Contemporary approaches have leveraged the potential of criticality metrics in various applications related to automated driving, e.g. for computationally assessing the dynamic risk or filtering large data sets to build scenario catalogs. As a prerequisite to systematically choose adequate criticality metrics for such applications, we extensively review the state of the art of criticality metrics, their properties, and their applications in the context of automated driving. Based on this review, we propose a suitability analysis as a methodical tool to be used by practitioners. Both the proposed method and the state of the art review can then be harnessed to select well-suited measurement tools that cover an application’s requirements, as demonstrated by an exemplary execution of the analysis. Ultimately, efficient, valid, and reliable measurements of an automated vehicle’s safety performance are a key requirement for demonstrating its trustworthiness.

show abstract

Section: Related Workmentioning

confidence: 99%

Criticality Metrics for Automated Driving: A Review and Suitability Analysis of the State of the Art

Westhofen

Neurohr

Koopmann

et al. 2022

Arch Computat Methods Eng

View full text Add to dashboard Cite

show abstract

“…In literature the TTC is defined as critical for values T T C crit ≤ 1.5s [43]. In addition to that a multidimensional criticality analysis as presented in [44] can also be applied. A scenario can be critical (a) or not (b) in the real test drive.…”

Section: E Credibility Assessmentmentioning

confidence: 99%

A Credibility Assessment Approach for Scenario-Based Virtual Testing of Automated Driving Functions

Stadler¹,

Montanari²,

Baron

et al. 2022

IEEE Open J. Intell. Transp. Syst.

Self Cite

View full text Add to dashboard Cite

An immense test space is pushing the development and testing of automated driving functions from real to virtual environments. The virtual world is provided by interconnected simulation models representing sensors, vehicle dynamics, and both static and dynamic environment. For the virtual validation of automated driving, special attention must be paid to the simulation's credibility, which can be impaired by inappropriate or inaccurate simulation models and tools. Therefore, in this work a method is proposed to assess the credibility of simulation-based testing for automated driving. The approach allows a qualitative and relatively quantitative comparisons between scenarios as well as between different simulation setups. Therefore, several uni-and multivariate metrics are applied towards a scoring of similarity of the behavior between simulation and real test drive. This is achieved by using ground truth data in form of simulation scenarios from real world measurement data. In this way, the virtual automated vehicle encounters the same conditions and surroundings than its counterpart in the real world for evaluating their similarity. The practical applicability of the proposed credibility assessment approach is demonstrated in a case study, in which the credibility of an exemplary simulation-based test bench is inferred.

show abstract

“…An initial overview on contemporary criticality metrics has been published by the authors and is extended and unified in this work [69, p. 14]. Similarly, there exists work within the AV community that refers to a certain subset of criticality metrics, but does not provide an extensive review on the possible choices [31,37]. Additionally, motion models play a crucial role when assessing future evaluations of a scene.…”

Section: Related Workmentioning

confidence: 99%

“…Test Evaluation (B.3.b) During or after the execution of a test case, the performance of the system under test needs to be evaluated. This can be done based on the previously defined pass-/fail-criteria, which involve -possibly multiple -criticality metrics indicating the AV's safety performance [37]. In contrast to defining pass-/fail-criteria, a test evaluation actually computes the value of metrics and compares their results against the given criteria.…”

Section: Testing (B3)mentioning

confidence: 99%

“…1. for pruning of an MRM for an ADF within an exemplary situation, a TTB ≥ 0.4s was used to classify braking as a viable MRM [95] (application A.3), 2. for scenario classification of car following and lane merging maneuvers, 1s was used as an estimate based on the human reaction time, while 0.6s was empirically observed as critical by human subjects [48] (application B.2), 3. for criticality-based scenario classification within simulation runs, 1s was used (in combination with other criticality metrics) [37] (application B.2).…”

Section: Properties Of the Ttb Metricmentioning

confidence: 99%

See 1 more Smart Citation

Criticality Metrics for Automated Driving: A Review and Suitability Analysis of the State of the Art

Westhofen¹,

Neurohr²,

Koopmann³

et al. 2021

Preprint

View full text Add to dashboard Cite

The large-scale deployment of automated vehicles on public roads has the potential to vastly change the transportation modalities of today's society. Although this pursuit has been initiated decades ago, there still exist open challenges in reliably ensuring that such vehicles operate safely in open contexts. While functional safety is a wellestablished concept, the question of measuring the behavioral safety of a vehicle remains subject to research. One way to both objectively and computationally analyze traffic conflicts is the development and utilization of so-called criticality metrics. Contemporary approaches have leveraged the potential of criticality metrics in various applications related to automated driving, e.g. for computationally assessing the dynamic risk or filtering large data sets to build scenario catalogs. As a prerequisite to systematically chooseThe research leading to these results is funded by the German Federal Ministry for Economic Affairs and Energy within the projects 'VVM -Verification & Validation Methods for Automated Vehicles Level 4 and 5' and 'SET Level -Simulation-based Development and Testing of Automated Driving', based on a decision by the Parliament of the Federal Republic of Germany.

show abstract

Evaluation of Virtual Traffic Situations for Testing Automated Driving Functions based on Multidimensional Criticality Analysis

Cited by 7 publications

References 12 publications

Criticality Metrics for Automated Driving: A Review and Suitability Analysis of the State of the Art

Criticality Metrics for Automated Driving: A Review and Suitability Analysis of the State of the Art

A Credibility Assessment Approach for Scenario-Based Virtual Testing of Automated Driving Functions

Criticality Metrics for Automated Driving: A Review and Suitability Analysis of the State of the Art

Contact Info

Product

Resources

About